Question: * Secret distribution via files https://github.com/GoogleCloudPlatform/kubernetes/pull/2030 * Docker secrets https://github.com/docker/docker/pull/6697 * Docker vault https://github.com/docker/docker/issues/10310 <ins> * Service Accounts: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/service_accounts.md * Secret volumes https://github.com/GoogleCloudPlatform/kubernetes/4126 </ins> ## Specific Design Points

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
/assign /area kubeadm
/sig cluster-lifecycle
i went trough all the flakes we have at the moment and adjusted a number timeouts. here is the PR:
was green for a bit since merge, but noticed a timeout on this morning. Do you think it is worth bumping the timeout there as well? It looks like the timeout was quite a large outlier compared to the successful runs (more than double the duration), so maybe we should address underlying problem.
lets wait for some more singlal before bumping the timeouts on init. locally for me takes a couple of minutes, so i would like to know if this is a temporary GCE/networking issue.
actually something is not right, because in this job: also this job showed that even 10m is not enough for it errored with: but looking at the log timestamps only ~2 minutes have passed. i will have another look later today or tomorrow.
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-master (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-master) /priority important-soon /remove-kind failing-test /kind flake the test seems to be timeouting occasionally now. this appeared to be a temporary network speed issue. in the meantime i have created this PR which attempts to catch network flakes early and reduce overall network traffic:
the kubeadm e2e jobs started requiring more than 40 minutes: https://k8s- so a lot of test jobs are now failing. this could be due to: resource limits in Prow more conformance tests were during the weekend or next week i will try bumping all jobs to ~60 minutes.
/remove-priority important-soon (prio is already critical)
xref
hi, the jobs seem to be now at the state where they were before the ticket was logged - i.e. mostly green. the Prow issue that was discussed in is definitely a factor, so i'm not going to bump the jobs to 60minutes for now.

we do still see 2 occasional types of flakes: one of them is known and we don't have a fix for it as it's complicated. for the other one i'm going to send a PR and ping you for a quick lgtm. thanks.
thanks for following up! I will make sure to take a look at your incoming PR :+1:
i haven't seen the same failure in the last 10 runs: https://k8s- please re-open this ticket if needed. thanks.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am seeing some pretty regular failures here again. Could you take a look? /reopen
Reopened this issue. <detailsIn response to : I am seeing some pretty regular failures here again. Could you take a look? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the job is taking more than 40 minutes. so this is related to:
The runs in last few days have been well under 60m, except for
i'm suspecting slow CPU/network for some of the recent failures.
/remove-sig testing SIG testing does not own kinder or upgrades
looks like the most prominent recent flakes are caused by the inability to prepare the node image or upgrade all nodes in less than 10 minutes. i will send a PR to increase these timeouts.
Is that test now named ?
yes. i see some occasional flakes of not being able to complete the conformance suite in 25 minutes (parallel).
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
i can conform that the suite requires mostly more than 25 minutes in later releases (either due to more tests or slower infra). this PR increases the e2e test suite in kubeadm to 35 minutes and will close this ticket: we should create new / separate tickets with flaky tasks in the job.


Question: []metav1.TableRow{{Cells: []interface{}{"test14", "2/2", "Running", "9 (5d ago)", "<unknown>"}}}, }, { <del> // Test PodScheduled condition with reason WaitingForGates </del> <ins> // Test PodScheduled condition with reason SchedulingGated </ins> api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "test15"}, Spec:       api.PodSpec{Containers: make([]api.Container, 2)},

<img width="844" alt="image" src="" <img width="942" alt="image" src="" Here are some incorrect comments and function name about SchedulingGated Comments and function names need to be changed to SchedulingGated nothing No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
/remove-kind bug /kind cleanup


Question: visible at runtime in the container. release: v1.9 file: test/e2e/common/downward_api.go <ins> - testname: PodTemplate lifecycle codename: '[sig-node] PodTemplates should run the lifecycle of PodTemplates [Conformance]' description: Attempt to create a PodTemplate. Patch the created PodTemplate. Fetching the PodTemplate MUST reflect changes. By fetching all the PodTemplates via a Label selector it MUST find the PodTemplate by it's static label and updated value. The PodTemplate must be deleted. release: v1.19 file: test/e2e/common/podtemplates.go </ins> - testname: LimitRange, resources codename: '[sig-scheduling] LimitRange should create a LimitRange with defaults and ensure pod has those defaults applied. [Conformance]'

This issue proposes the writing of a new test for the API endponts , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these results from APIsnoop, we are lacking in test coverage and conformant tests for PodTemplate. We chose these three following endpoints for this mock test: - - Outline: PodTemplates, Expect no PodTemplates a PodTemplate (1), Expect no error PodTemplate (1), Ensure the same as how it was created PodTemplate (1) with label a=1, Expect no error PodTemplate (1), Expect label in returned data to be present PodTemplates by label a=1, Expect one PodTemplate PodTemplate (1) Expect no error PodTemplates, Expect no PodTemplates Fetch dependencies: This mock test is using client-go's standard functions to talk to the PodTemplate part of the API. If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
We decided not to deprecate. We can link to that, but otherwise this is OK to write / promote.
ref: for the decision not to deprecate
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: request.VolumeType = aws.String(createType) request.Encrypted = aws.Bool(volumeOptions.Encrypted) if len(volumeOptions.KmsKeyId) > 0 { <del> if missing, err := c.checkEncryptionKey(volumeOptions.KmsKeyId); err != nil { if missing { // KSM key is missing, provisioning would fail return "", err } // Log checkEncryptionKey error and try provisioning anyway. glog.Warningf("Cannot check KSM key %s: %v", volumeOptions.KmsKeyId, err) } </del> request.KmsKeyId = aws.String(volumeOptions.KmsKeyId) request.Encrypted = aws.Bool(true) }

This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events.

Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV.

And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available".

I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:


Question: "fmt" "io/ioutil" <ins> "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/runtime/serializer" </ins> auditinternal "k8s.io/apiserver/pkg/apis/audit" auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" "k8s.io/apiserver/pkg/apis/audit/validation" "k8s.io/apiserver/pkg/audit" "k8s.io/klog/v2" )

Currently the [audit policy decoder][1] silently drops unknown fields. For backwards compatibility, we probably cannot fail on unknown fields, but it would be good to at least log a warning when they're encountered. [1]: /sig api-machinery
/triage accepted
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm not familiar with apiserver, but this seems like a simple one. /assign
How can we find unknown fields in the yaml file? Currently I'm think of converting both the yaml file and the decoded Policy struct to two map[string]interface{}, and compare both map to find out which keys are in the yaml file but not in the decoded struct, is there any better solution?
Courtesy on slack:


Question: framework.Logf("Pod %s on node %s %s total=%s run=%s execute=%s", pod.Name, pod.Spec.NodeName, scenario, total, v.completeDuration, v.duration) return pod, errs } <del> // findContainerStatusInPod finds a container status by its name in the provided pod func findContainerStatusInPod(pod *v1.Pod, containerName string) *v1.ContainerStatus { for _, container := range pod.Status.InitContainerStatuses { if container.Name == containerName { return &container } } for _, container := range pod.Status.ContainerStatuses { if container.Name == containerName { return &container } } for _, container := range pod.Status.EphemeralContainerStatuses { if container.Name == containerName { return &container } } return nil } </del>

We would like to know early if the behavior is broken on any container runtime implementation. This is motivated by the discussion under the PR:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: importpath = "k8s.io/apimachinery/pkg/util/yaml", deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/json:go_default_library", <del> "//vendor/k8s.io/klog/v2:go_default_library", </del> "//vendor/sigs.k8s.io/yaml:go_default_library", ], )

This bug was privately disclosed and told to open here. What happened: NewYAMLOrJSONDecoder from has an index out of bounds issue which can cause callers to panic. Download attached reproducer file Run golang test: When run with v0.19.3: Additionally, this can be seen in kubectl directly (below is kubectl 1.20.0): Environment: Kubernetes version (use ): Verified with 1.19, 1.20
/sig api-machinery
This piece attempts to find line+column position of a json syntax error. To do that it assumes that full input is available which is not true: is : is only the first data chunk: The subject happens e.g. when syntax error offset is past buffer size and Test to reproduce the panic: I am not sure it is easily possible to find line+column of json syntax error as full input may not be available. Maybe it should only attempt to find line+column if offset is within the first chunk (i.e. below buffer size).
Line number calculation in and updated in
do you want to submit a patch to with a defensive check to avoid the panic ( compare and )
I have raised a discussion on having line number in the . The consensus there is that it should be possible to count lines with a full input at hand and it does not worth to have it in the core library due to performance impact and maintenance costs. The problem of here is that it attempts to find out line number without having full input at hand. I think the defensive patch is not the right approach because line calculation is incorrect anyway even when there is no panic: begins at a value with a syntax error, i.e. in a stream of two values and syntax error in the second value it would contain only the second value and thus new lines in the first value in the stream would not be counted. I think the following options are there: to stream taken from . This is straightforward but changes interface. I can not find any code that uses field, it is only used in the error message that would change: line number only if offset is within the length of the first data chunk (). I am not sure what would be the unknown line number then, perhaps. line number properly during stream decoding, see This has some performance penalty as line number is calculated in any case even if there is no syntax error, i.e.

in the majority of the cases.
cc /triage accepted
This is probably acceptable.
I have opened a PR
Thank you for support.


Question: "net/url" "os" "reflect" <ins> "runtime" "strings" </ins> "testing" "k8s.io/kubernetes/pkg/util/sets"

When using go 1.7, is failing with the following error:
Emm... I'll need this fix
Wouldn't a full struct copy work if we zero out the fields we don't want retained afterwards? It'll be less fragile as well.
You prefer to re-open my pull request or include that commit in ?
I'm also trouble in the issue when I execute command on release-1.4 branch of Kubernetes. Have someone fixed it? Fails (all fails in ):
, we can not just re-open : in , it did not include & which means build failed in go1.6 if re-open .
BTW, do we still support go1.6 in master? If so, we need upmerge to master.
Is this still a problem? I don't see any test failures on go-1.7.3

Hey Guys, When running command on release-1.4 branch, I got the below errors, my GO version is , how to fix it or something else have I missed?
dup of . fixed by . we can close this. /sig api-machinery
Sure, close it.


Question: // translateMountPropagation transforms v1.MountPropagationMode to // runtimeapi.MountPropagation. func translateMountPropagation(mountMode *v1.MountPropagationMode) (runtimeapi.MountPropagation, error) { <ins> if runtime.GOOS == "windows" { // Windows containers doesn't support mount propagation, use private for it. // Refer https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation. return runtimeapi.MountPropagation_PROPAGATION_PRIVATE, nil } </ins> if !utilfeature.DefaultFeatureGate.Enabled(features.MountPropagation) { // mount propagation is disabled, use private as in the old versions return runtimeapi.MountPropagation_PROPAGATION_PRIVATE, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Windows containers creation failed because of rslave mounts. What you expected to happen: Windows containers don't support mount propagation, we should disable it for windows. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows
What's your pod config for mount?
if you agree with this as a blocker for 1.10, could you go ahead and add "approved-for-milestone" label to this, as well as status? That will help it stay in the milestone.
/status approved-for-milestone
Any pods on Windows would meet same issue. It defaults to rslave now on master branch.


Question: // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. It("should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters]", func() { <ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. framework.SkipIfProviderIs("aws") </ins> svc := getServeHostnameService("service") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: The test should either succeed or be skipped on AWS How to reproduce it (as minimally and precisely as possible): Run this test on an AWS cluster with the AWS cloud provider configured Anything else we need to know?: The test sets the service's session affinity to here: The AWS cloud provider specifically only supports : Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/cc
Thanks for the investigation, given the information so far, we should just skip this test on AWS. Will prepare a PR soon.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/component-base/metrics/testutil"

While validating nftable mode using conformance test, the following error logs were produced consistently: I thought it was because a known issue described here: However, the error was produced for every run and every kube-proxy instance, which doesn't seem related to the kernel bug. After some investigation, I think it's caused by a bug: in some cases a chain could change from stale to active, but once it's to it will always be deleted once. When the proxier tries to delete a previously stale but currently active chain, it would fail and lead to these errors, though it won't cause real problem thanks to kernel's validation. cc kube-proxy shouldn't try to delete active chains and produce such error logs. The e2e test case can reproduce it consistantly: No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I have a PR to fix it.
/sig network /area proxy
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /area proxy </details


Question: ss, err := newTestScaleSet(ctrl) assert.NoError(t, err, test.description) <del> expectedVMSS := buildTestVMSS([]string{testLBBackendpoolID0}) </del> <ins> expectedVMSS := buildTestVMSS(testVMSSName, []string{testLBBackendpoolID0}) </ins> mockVMSSClient := ss.cloud.VirtualMachineScaleSetsClient.(*mockvmssclient.MockInterface) mockVMSSClient.EXPECT().List(gomock.Any(), ss.ResourceGroup).Return([]compute.VirtualMachineScaleSet{expectedVMSS}, nil).AnyTimes() mockVMSSClient.EXPECT().Get(gomock.Any(), ss.ResourceGroup, testVMSSName).Return(expectedVMSS, nil).MaxTimes(1)

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: var _ = SIGDescribe("SubjectReview", func() { f := framework.NewDefaultFramework("subjectreview") <del> ginkgo.It("should support SubjectReview API operations", func() { </del> <ins> /* Release: v1.27 Testname: SubjectReview, API Operations Description: A ServiceAccount is created which MUST succeed. A clientset is created to impersonate the ServiceAccount. A SubjectAccessReview is created for the ServiceAccount which MUST succeed. The allowed status for the SubjectAccessReview MUST match the expected allowed for the impersonated client call. A LocalSubjectAccessReviews is created for the ServiceAccount which MUST succeed. The allowed status for the LocalSubjectAccessReview MUST match the expected allowed for the impersonated client call. */ framework.ConformanceIt("should support SubjectReview API operations", func() { </ins> AuthClient := f.ClientSet.AuthorizationV1() ns := f.Namespace.Name

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 31 Dec 2022 [X] test promotion pr : According to following two APIsnoop queries, there are still two authorization endpoints which are untested. example endpoint kind ---------------------------------------------------------+
-+-------------------------- createAuthorizationV1NamespacedLocalSubjectAccessReview LocalSubjectAccessReview (1 row) sql-mode select distinct endpoint, testhit AS "e2e Test", conftesthit AS "Conformance Test" from public.auditevent where endpoint ilike '%SubjectAccessReview' and useragent like '%e2e%' and not conftesthit order by endpoint limit 10; example endpoint Conformance Test ------------------------------------------+----------+------------------ createAuthorizationV1SubjectAccessReview f (1 row) [It] should support SubjectReview API operations STEP: Create pod "pod-x6fhh" in namespace "subjectreview-1704" 12/07/22 21:33:12.332 Dec 7 21:33:12.441: INFO: Waiting up to 5m0s for pod "pod-x6fhh" in namespace "subjectreview-1704" to be "running" Dec 7 21:33:12.502: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 61.265222ms Dec 7 21:33:14.512: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 2.070783878s Dec 7 21:33:16.520: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 4.078677494s Dec 7 21:33:18.513: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 6.07168643s Dec 7 21:33:20.510: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 8.068598657s Dec 7 21:33:22.509: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false.

Elapsed: 10.068442532s Dec 7 21:33:24.514: INFO: Pod "pod-x6fhh": Phase="Pending", Reason="", readiness=false. Elapsed: 12.073076728s Dec 7 21:33:26.511: INFO: Pod "pod-x6fhh": Phase="Running", Reason="", readiness=true. Elapsed: 14.069523123s Dec 7 21:33:26.511: INFO: Pod "pod-x6fhh" satisfied condition "running" Dec 7 21:33:26.515: INFO: "pod-x6fhh" in namespace "subjectreview-1704" is "Running" Dec 7 21:33:26.515: INFO: serviceaccount name: "system:serviceaccount:subjectreview-1704:default" STEP: Creating SubjectAccessReview in "subjectreview-1704" namespace 12/07/22 21:33:26.515 Dec 7 21:33:26.521: INFO: sarResponse Status: v1.SubjectAccessReviewStatus{Allowed:false, Denied:false, Reason:"", EvaluationError:""} STEP: Creating clientset to impersonate "system:serviceaccount:subjectreview-1704:default" 12/07/22 21:33:26.521 STEP: Verifying api 'get' call to "pod-x6fhh" as "system:serviceaccount:subjectreview-1704:default" 12/07/22 21:33:26.522 Dec 7 21:33:26.525: INFO: api call by "system:serviceaccount:subjectreview-1704:default" was denied Dec 7 21:33:26.525: INFO: SubjectAccessReview has been verified STEP: Creating a LocalSubjectAccessReview in "subjectreview-1704" namespace 12/07/22 21:33:26.525 Dec 7 21:33:26.530: INFO: lsarResponse Status: v1.SubjectAccessReviewStatus{Allowed:false, Denied:false, Reason:"", EvaluationError:""} STEP: Verifying api 'get' call to "pod-x6fhh" as "system:serviceaccount:subjectreview-1704:default" 12/07/22 21:33:26.

53 Dec 7 21:33:26.532: INFO: api call by "system:serviceaccount:subjectreview-1704:default" was denied Dec 7 21:33:26.532: INFO: LocalSubjectAccessReview has been verified sql-mode select distinct substring(endpoint from 'w+') AS endpoint, right(useragent,43) AS useragent from testing.auditevent where useragent like 'e2e%should%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)*1000,0) - and endpoint ilike '%subject%' order by endpoint limit 10; example endpoint | useragent ---------------------------------------------------------+--------------------------------------------- createAuthorizationV1NamespacedLocalSubjectAccessReview | should support SubjectReview API operations createAuthorizationV1SubjectAccessReview | should support SubjectReview API operations (2 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.PodNominator stop  chan struct{} <del> clock util.Clock </del> <ins> clock clock.Clock </ins> // pod initial backoff duration. podInitialBackoffDuration time.Duration

We should use /kind cleanup
/remove-kind feature /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-kind feature /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /triage accepted
/assign


Question: args = append(args, fmt.Sprintf("--image-type=%s", img)) } _, _, err := RunCmd("gcloud", args...) <del> return err </del> <ins> if err != nil { return err } waitForSSHTunnels() return nil </ins> } // CheckNodesReady waits up to nt for expect nodes accessed by c to be ready,

Upgrade fails with: Error from: https://k8s- Related failures: https://k8s- https://k8s-
/cc
might require how do the GKE tests interact with the defaults you set up?
Looks like are pretty isolated to upgrade tests.
Also we didn't start seeing this until the ABAC default change merged so I think change might help
it will definitely help with the GCE jobs mentioned in the description of
I haven't looked at the logs, so sorry if this is just noise, but the symptom (couldn't get logs b/c pod doesn't exist) looks very similar to what happened when the kubectl taint e2e tests were running in parallel instead of serial.
Judging by the comment and digging into the history , this used to be wrapped in a retry loop. It was removed in PR
it looks like this is limited to gke jobs in the upgrade tests. since getting pod logs depends on master-node communications, it would be affected by the SSH tunnels issue ()
Not blocking under guidance


Question: ARG BASEIMAGE FROM $BASEIMAGE <del> LABEL image_version="1.29" </del> <ins> LABEL image_version="1.29-1" </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted


Question: } } <del> // Test_RemoveNodeFromAttachUpdates_Positive expects an entire node entry to be removed // from nodesToUpdateStatusFor func Test_RemoveNodeFromAttachUpdates_Positive(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates(nodeName) // Assert if err != nil { t.Fatalf("RemoveNodeFromAttachUpdates should not return error, but got: %v", err) } if len(asw.nodesToUpdateStatusFor) > 0 { t.Fatal("nodesToUpdateStatusFor should be empty as its only entry has been deleted.") } } // Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist expects an error to be thrown // when nodeName is not in nodesToUpdateStatusFor. func Test_RemoveNodeFromAttachUpdates_Negative_NodeDoesNotExist(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.RemoveNodeFromAttachUpdates("node-2") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as the nodeName doesn't exist.") } if len(asw.nodesToUpdateStatusFor) != 1 { t.Fatal("The length of nodesToUpdateStatusFor should not change because no operation was performed.") } } // Test_RemoveNodeFromAttachUpdates_Negative_Empty expects an error to be thrown // when a nodesToUpdateStatusFor is empty. func Test_RemoveNodeFromAttachUpdates_Negative_Empty(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } // Act err := asw.RemoveNodeFromAttachUpdates("node-1") // Assert if err == nil { t.Fatal("RemoveNodeFromAttachUpdates should return an error as nodeToUpdateStatusFor is empty.") } if len(asw.nodesToUpdateStatusFor) != 0 { t.Fatal("The length of nodesToUpdateStatusFor should be 0.") } } </del> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: Deployment running with read-only GCE PD. A node containing one of the deployment pods is deleted and recreated soon after. New pod is scheduled to the same node, but is stuck in ContainerCreating state. The node's kubelet logs show continuous "Volume is not yet attached according to node" errors. What you expected to happen: Pod recreated successfully. Anything else we need to know?: Cause by PR The issue happens as follows: When the node is deleted, GCE PD is detached from the cloud provider. AttachDetachController discovers the PD has been detached, and marks it as detached. The original pod still exists in the API server (i.e. before garbage collection), so AttachDetachController reattaches the PD, successfully. Before the node recovers, ADC tries to update the node status but finds that the node doesn't exist. Following the logic introduced in it never attempts to update the node status again. The node recovers, a new pod is scheduled to it, waits for a successful attach signal in the node status but never gets it. /cc /sig storage
/assign


Question: }) }) <del> var _ = SIGDescribe("ESIPP [Slow]", func() { f := framework.NewDefaultFramework("esipp") var loadBalancerCreateTimeout time.Duration var cs clientset.Interface serviceLBNames := []string{} ginkgo.BeforeEach(func() { // requires cloud load-balancer support - this feature currently supported only on GCE/GKE e2eskipper.SkipUnlessProviderIs("gce", "gke") cs = f.ClientSet loadBalancerCreateTimeout = e2eservice.GetServiceLoadBalancerCreationTimeout(cs) }) ginkgo.AfterEach(func() { if ginkgo.CurrentGinkgoTestDescription().Failed { DescribeSvc(f.Namespace.Name) } for _, lb := range serviceLBNames { framework.Logf("cleaning load balancer resource for %s", lb) e2eservice.CleanupServiceResources(cs, lb, framework.TestContext.CloudConfig.Region, framework.TestContext.CloudConfig.Zone) } //reset serviceLBNames serviceLBNames = []string{} }) ginkgo.It("should work for type=LoadBalancer", func() { namespace := f.Namespace.Name serviceName := "external-local-lb" jig := e2eservice.NewTestJig(cs, namespace, serviceName) svc, err := jig.CreateOnlyLocalLoadBalancerService(loadBalancerCreateTimeout, true, nil) framework.ExpectNoError(err) serviceLBNames = append(serviceLBNames, cloudprovider.DefaultLoadBalancerName(svc)) healthCheckNodePort := int(svc.Spec.HealthCheckNodePort) if healthCheckNodePort == 0 { framework.Failf("Service HealthCheck NodePort was not allocated") } defer func() { err = jig.ChangeServiceType(v1.ServiceTypeClusterIP, loadBalancerCreateTimeout) framework.ExpectNoError(err) // Make sure we didn't leak the health check node port. const threshold = 2 nodes, err := getEndpointNodesWithInternalIP(jig) framework.ExpectNoError(err) config := e2enetwork.NewNetworkingTestConfig(f) for _, internalIP := range nodes { err := testHTTPHealthCheckNodePortFromTestContainer( config, internalIP, healthCheckNodePort, e2eservice.KubeProxyLagTimeout, false, threshold) framework.ExpectNoError(err) } err = cs.CoreV1().Services(svc.Namespace).Delete(context.TODO(), svc.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() svcTCPPort := int(svc.Spec.Ports[0].Port) ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) ginkgo.By("reading clientIP using the TCP service's service port via its external VIP") clientIP, err := GetHTTPContent(ingressIP, svcTCPPort, e2eservice.KubeProxyLagTimeout, "/clientip") framework.ExpectNoError(err) framework.Logf("ClientIP detected by target pod using VIP:SvcPort is %s", clientIP) ginkgo.By("checking if Source IP is preserved") if strings.HasPrefix(clientIP, "10.") { framework.Failf("Source IP was NOT preserved") } }) ginkgo.It("should work for type=NodePort", func() { namespace := f.Namespace.Name serviceName := "external-local-nodeport" jig := e2eservice.NewTestJig(cs, namespace, serviceName) svc, err := jig.CreateOnlyLocalNodePortService(true) framework.ExpectNoError(err) defer func() { err := cs.CoreV1().Services(svc.Namespace).Delete(context.TODO(), svc.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() tcpNodePort := int(svc.Spec.Ports[0].NodePort) endpointsNodeMap, err := getEndpointNodesWithInternalIP(jig) framework.ExpectNoError(err) dialCmd := "clientip" config := e2enetwork.NewNetworkingTestConfig(f) for nodeName, nodeIP := range endpointsNodeMap { ginkgo.By(fmt.Sprintf("reading clientIP using the TCP service's NodePort, on node %v: %v:%v/%v", nodeName, nodeIP, tcpNodePort, dialCmd)) clientIP, err := GetHTTPContentFromTestContainer(config, nodeIP, tcpNodePort, e2eservice.KubeProxyLagTimeout, dialCmd) framework.ExpectNoError(err) framework.Logf("ClientIP detected by target pod using NodePort is %s, the ip of test container is %s", clientIP, config.TestContainerPod.Status.PodIP) // the clientIP returned by agnhost contains port if !strings.HasPrefix(clientIP, config.TestContainerPod.Status.PodIP) { framework.Failf("Source IP was NOT preserved") } } }) ginkgo.It("should only target nodes with endpoints", func() { namespace := f.Namespace.Name serviceName := "external-local-nodes" jig := e2eservice.NewTestJig(cs, namespace, serviceName) nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, e2eservice.MaxNodesForEndpointsTests) framework.ExpectNoError(err) svc, err := jig.CreateOnlyLocalLoadBalancerService(loadBalancerCreateTimeout, false, func(svc *v1.Service) { // Change service port to avoid collision with opened hostPorts // in other tests that run in parallel. if len(svc.Spec.Ports) != 0 { svc.Spec.Ports[0].TargetPort = intstr.FromInt(int(svc.Spec.Ports[0].Port)) svc.Spec.Ports[0].Port = 8081 } }) framework.ExpectNoError(err) serviceLBNames = append(serviceLBNames, cloudprovider.DefaultLoadBalancerName(svc)) defer func() { err = jig.ChangeServiceType(v1.ServiceTypeClusterIP, loadBalancerCreateTimeout) framework.ExpectNoError(err) err := cs.CoreV1().Services(svc.Namespace).Delete(context.TODO(), svc.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() healthCheckNodePort := int(svc.Spec.HealthCheckNodePort) if healthCheckNodePort == 0 { framework.Failf("Service HealthCheck NodePort was not allocated") } ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) svcTCPPort := int(svc.Spec.Ports[0].Port) const threshold = 2 config := e2enetwork.NewNetworkingTestConfig(f) for i := 0; i < len(nodes.Items); i++ { endpointNodeName := nodes.Items[i].Name ginkgo.By("creating a pod to be part of the service " + serviceName + " on node " + endpointNodeName) _, err = jig.Run(func(rc *v1.ReplicationController) { rc.Name = serviceName if endpointNodeName != "" { rc.Spec.Template.Spec.NodeName = endpointNodeName } }) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("waiting for service endpoint on node %v", endpointNodeName)) err = jig.WaitForEndpointOnNode(endpointNodeName) framework.ExpectNoError(err) // HealthCheck should pass only on the node where num(endpoints) > 0 // All other nodes should fail the healthcheck on the service healthCheckNodePort for n, internalIP := range ips { // Make sure the loadbalancer picked up the health check change. // Confirm traffic can reach backend through LB before checking healthcheck nodeport. e2eservice.TestReachableHTTP(ingressIP, svcTCPPort, e2eservice.KubeProxyLagTimeout) expectedSuccess := nodes.Items[n].Name == endpointNodeName port := strconv.Itoa(healthCheckNodePort) ipPort := net.JoinHostPort(internalIP, port) framework.Logf("Health checking %s, http://%s/healthz, expectedSuccess %v", nodes.Items[n].Name, ipPort, expectedSuccess) err := testHTTPHealthCheckNodePortFromTestContainer( config, internalIP, healthCheckNodePort, e2eservice.KubeProxyEndpointLagTimeout, expectedSuccess, threshold) framework.ExpectNoError(err) } framework.ExpectNoError(e2erc.DeleteRCAndWaitForGC(f.ClientSet, namespace, serviceName)) } }) ginkgo.It("should work from pods", func() { var err error namespace := f.Namespace.Name serviceName := "external-local-pods" jig := e2eservice.NewTestJig(cs, namespace, serviceName) svc, err := jig.CreateOnlyLocalLoadBalancerService(loadBalancerCreateTimeout, true, nil) framework.ExpectNoError(err) serviceLBNames = append(serviceLBNames, cloudprovider.DefaultLoadBalancerName(svc)) defer func() { err = jig.ChangeServiceType(v1.ServiceTypeClusterIP, loadBalancerCreateTimeout) framework.ExpectNoError(err) err := cs.CoreV1().Services(svc.Namespace).Delete(context.TODO(), svc.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) port := strconv.Itoa(int(svc.Spec.Ports[0].Port)) ipPort := net.JoinHostPort(ingressIP, port) path := fmt.Sprintf("%s/clientip", ipPort) ginkgo.By("Creating pause pod deployment to make sure, pausePods are in desired state") deployment := createPausePodDeployment(cs, "pause-pod-deployment", namespace, 1) framework.ExpectNoError(e2edeployment.WaitForDeploymentComplete(cs, deployment), "Failed to complete pause pod deployment") defer func() { framework.Logf("Deleting deployment") err = cs.AppsV1().Deployments(namespace).Delete(context.TODO(), deployment.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "Failed to delete deployment %s", deployment.Name) }() deployment, err = cs.AppsV1().Deployments(namespace).Get(context.TODO(), deployment.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Error in retrieving pause pod deployment") labelSelector, err := metav1.LabelSelectorAsSelector(deployment.Spec.Selector) framework.ExpectNoError(err, "Error in setting LabelSelector as selector from deployment") pausePods, err := cs.CoreV1().Pods(namespace).List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector.String()}) framework.ExpectNoError(err, "Error in listing pods associated with pause pod deployments") pausePod := pausePods.Items[0] framework.Logf("Waiting up to %v curl %v", e2eservice.KubeProxyLagTimeout, path) cmd := fmt.Sprintf(`curl -q -s --connect-timeout 30 %v`, path) var srcIP string loadBalancerPropagationTimeout := e2eservice.GetServiceLoadBalancerPropagationTimeout(cs) ginkgo.By(fmt.Sprintf("Hitting external lb %v from pod %v on node %v", ingressIP, pausePod.Name, pausePod.Spec.NodeName)) if pollErr := wait.PollImmediate(framework.Poll, loadBalancerPropagationTimeout, func() (bool, error) { stdout, err := framework.RunHostCmd(pausePod.Namespace, pausePod.Name, cmd) if err != nil { framework.Logf("got err: %v, retry until timeout", err) return false, nil } srcIP = strings.TrimSpace(strings.Split(stdout, ":")[0]) return srcIP == pausePod.Status.PodIP, nil }); pollErr != nil { framework.Failf("Source IP not preserved from %v, expected '%v' got '%v'", pausePod.Name, pausePod.Status.PodIP, srcIP) } }) ginkgo.It("should handle updates to ExternalTrafficPolicy field", func() { namespace := f.Namespace.Name serviceName := "external-local-update" jig := e2eservice.NewTestJig(cs, namespace, serviceName) nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, e2eservice.MaxNodesForEndpointsTests) framework.ExpectNoError(err) if len(nodes.Items) < 2 { framework.Failf("Need at least 2 nodes to verify source ip from a node without endpoint") } svc, err := jig.CreateOnlyLocalLoadBalancerService(loadBalancerCreateTimeout, true, nil) framework.ExpectNoError(err) serviceLBNames = append(serviceLBNames, cloudprovider.DefaultLoadBalancerName(svc)) defer func() { err = jig.ChangeServiceType(v1.ServiceTypeClusterIP, loadBalancerCreateTimeout) framework.ExpectNoError(err) err := cs.CoreV1().Services(svc.Namespace).Delete(context.TODO(), svc.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() // save the health check node port because it disappears when ESIPP is turned off. healthCheckNodePort := int(svc.Spec.HealthCheckNodePort) ginkgo.By("turning ESIPP off") svc, err = jig.UpdateService(func(svc *v1.Service) { svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster }) framework.ExpectNoError(err) if svc.Spec.HealthCheckNodePort > 0 { framework.Failf("Service HealthCheck NodePort still present") } epNodes, err := jig.ListNodesWithEndpoint() framework.ExpectNoError(err) // map from name of nodes with endpoint to internal ip // it is assumed that there is only a single node with the endpoint endpointNodeMap := make(map[string]string) // map from name of nodes without endpoint to internal ip noEndpointNodeMap := make(map[string]string) for _, node := range epNodes { ips := e2enode.GetAddresses(&node, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", node.Name) } endpointNodeMap[node.Name] = ips[0] } for _, n := range nodes.Items { ips := e2enode.GetAddresses(&n, v1.NodeInternalIP) if len(ips) < 1 { framework.Failf("No internal ip found for node %s", n.Name) } if _, ok := endpointNodeMap[n.Name]; !ok { noEndpointNodeMap[n.Name] = ips[0] } } framework.ExpectNotEqual(len(endpointNodeMap), 0) framework.ExpectNotEqual(len(noEndpointNodeMap), 0) svcTCPPort := int(svc.Spec.Ports[0].Port) svcNodePort := int(svc.Spec.Ports[0].NodePort) ingressIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) path := "/clientip" dialCmd := "clientip" config := e2enetwork.NewNetworkingTestConfig(f) ginkgo.By(fmt.Sprintf("endpoints present on nodes %v, absent on nodes %v", endpointNodeMap, noEndpointNodeMap)) for nodeName, nodeIP := range noEndpointNodeMap { ginkgo.By(fmt.Sprintf("Checking %v (%v:%v/%v) proxies to endpoints on another node", nodeName, nodeIP[0], svcNodePort, dialCmd)) _, err := GetHTTPContentFromTestContainer(config, nodeIP, svcNodePort, e2eservice.KubeProxyLagTimeout, dialCmd) framework.ExpectNoError(err, "Could not reach HTTP service through %v:%v/%v after %v", nodeIP, svcNodePort, dialCmd, e2eservice.KubeProxyLagTimeout) } for nodeName, nodeIP := range endpointNodeMap { ginkgo.By(fmt.Sprintf("checking kube-proxy health check fails on node with endpoint (%s), public IP %s", nodeName, nodeIP)) var body string pollFn := func() (bool, error) { // we expect connection failure here, but not other errors resp, err := config.GetResponseFromTestContainer( "http", "healthz", nodeIP, healthCheckNodePort) if err != nil { return false, nil } if len(resp.Errors) > 0 { return true, nil } if len(resp.Responses) > 0 { body = resp.Responses[0] } return false, nil } if pollErr := wait.PollImmediate(framework.Poll, e2eservice.TestTimeout, pollFn); pollErr != nil { framework.Failf("Kube-proxy still exposing health check on node %v:%v, after ESIPP was turned off. body %s", nodeName, healthCheckNodePort, body) } } // Poll till kube-proxy re-adds the MASQUERADE rule on the node. ginkgo.By(fmt.Sprintf("checking source ip is NOT preserved through loadbalancer %v", ingressIP)) var clientIP string pollErr := wait.PollImmediate(framework.Poll, e2eservice.KubeProxyLagTimeout, func() (bool, error) { clientIP, err := GetHTTPContent(ingressIP, svcTCPPort, e2eservice.KubeProxyLagTimeout, "/clientip") if err != nil { return false, nil } if strings.HasPrefix(clientIP, "10.") { return true, nil } return false, nil }) if pollErr != nil { framework.Failf("Source IP WAS preserved even after ESIPP turned off. Got %v, expected a ten-dot cluster ip.", clientIP) } // TODO: We need to attempt to create another service with the previously // allocated healthcheck nodePort. If the health check nodePort has been // freed, the new service creation will succeed, upon which we cleanup. // If the health check nodePort has NOT been freed, the new service // creation will fail. ginkgo.By("setting ExternalTraffic field back to OnlyLocal") svc, err = jig.UpdateService(func(svc *v1.Service) { svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal // Request the same healthCheckNodePort as before, to test the user-requested allocation path svc.Spec.HealthCheckNodePort = int32(healthCheckNodePort) }) framework.ExpectNoError(err) pollErr = wait.PollImmediate(framework.Poll, e2eservice.KubeProxyLagTimeout, func() (bool, error) { clientIP, err := GetHTTPContent(ingressIP, svcTCPPort, e2eservice.KubeProxyLagTimeout, path) if err != nil { return false, nil } ginkgo.By(fmt.Sprintf("Endpoint %v:%v%v returned client ip %v", ingressIP, svcTCPPort, path, clientIP)) if !strings.HasPrefix(clientIP, "10.") { return true, nil } return false, nil }) if pollErr != nil { framework.Failf("Source IP (%v) is not the client IP even after ESIPP turned on, expected a public IP.", clientIP) } }) }) </del> // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of // affinity test for non-load-balancer services. Session afinity will be // enabled when the service is created and a short timeout will be configured so

Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: Services should be able to change the type and ports of a service [Slow] Testgrid link: https://k8s- Reason for failure: Test timeouts in ~half of the runs since it was enabled here: /sig scalability /sig network
/cc /cc
The context is - the problems are strictly related to scale - 5k-node jobs are timing out. I very briefly skimmed through the test and didn't see anything obviously suspicious (apart from the fact there are gazillions long timeouts there, which alltogether sum up long hours). That said - it seems to me that there are effectively two separate tests inside it: one for TCP one for UDP They don't seem to share anything with each other. - would it make sense to split this test two effectively two separate tests: one for TCP and one for UDP. (I guess it would also allow us to share some code there).
[And potentially it could help with debugging the problem if it is strictly related to only one of those cases...]
I wanted to focus on this past week but I didn´t have time, this is my list of more expensive times and approx average duration, all of the sig-network ones use LoadBalancers. I wanted to time the test to understand if there are places where can save time ... I don´t have much context on those tests, maybe Tim can help us more, but this comment makes me think that the use of both protocols is intended
/triage accepted
/triage accepted It's been a very long time since I touched this, and I recall holding my nose on it in the first place. Splitting TCP from UDP seems reasonable if we can run them in parallel. we'll want to share the flow, and parameterize some shared funcs. I don't know that this will prevent test timeouts, though.
We could use a volunteer to go poke at this, and see how hard it will really be.
You mean "forcing them to run in parallel"? Or "allowing them to run in parallel"? The later doesn't require any additiona work. The former I think mean that it will still have to be a single test (just the code will be organized better) and that won't really help with debugging (nor preventing timeouts).

this was discussed yesterday in sig-net meeting and my understanding was to split current test in 2 different so the test can run in parallel, if you want to do so and one protocol doesn´t have to wait for the other. I think that is what you meant here
As long as we CAN parallelize. Creating LBs can be sloooooow. :P Antonio Ojea <:
Reopening - I don't think just splitting will address the problem. It may help us with debugging though...
With the split we moved from 60 mins to run the test, to run 2 tests in [30-40] mins each https://k8s- Seems a win :smile:
Awesome - that's something I was silently counting for TBH :) [I doubt it will fix flakiness, but maybe it will make it easier to debug...]
Actually - I think we're getting close thanks to the two changes ( + this one). Here is the run: This particular one filed on UDP (not sure if it's just UDP-related, or just manifested with UDP now). And we have a clear failure now: In my PR I decreased the timeout from 2h to 45m, as it seemed to me that in the past it should be enough. But it failed on 45m timeout here. So I think the next question now is: is it that slow - that we need more than 45m or something isn't working as expected (either in the k8s itself or in the test) But I think we have a clear single place to look at now.
Actually - the full stacktrace: So it's this line that failed: What is interesting is that something was changing during those 45 minutes:
Note that it's 4th time we actually wait for LB in this test. The failure happens after the pod were scaled down to 0 and then to 1. So that makes me think, that's it's actually not LB-related, but rather something internally in k8s, becuase L4 LB doesn't depend on pods - it targets nodes...
If it is always UDP, I start to see ghosts between GCE and UDP ... we only have KIND to compare against, or do we have some other non gce enviroment?

I mean, there are other tests failing in UDP , and show much more failures in GCE than in kind
- this does not look "normal". Can we give some prio to helping track this flake?
hmm interesting. we will investigate.
/assign
- any progress on this?
I haven't had a chance to look into this yet. I will prioritize it this week.
Digging through the logs I believe this was a bug in the test code that was fixed by . In the test, the service port changes around 12:33. However due to the bug, the test continues to use the old service port as seen in the test output: The port should be 81. I believe the following sequence occurs from this point: 1) The reachability test fails in the beginning because the service has changed the port and the load balancer has not been updated yet resulting in the error. 1) Around 12:38, the service controller is syncing the service and notices that the service port has changed and starts to delete the forwarding rule causing the error to change to . 1) Once the forwarding rule is recreated, the error goes back to being because the forwarding rule exists but the wrong port is used. Looking at , the test runs after the one linked have all passed. Is there anything else that needs to be investigated as part of this issue?
OK - the test was quite flaky in the past but indeed it didn't flake for the last 3 weeks (neither this nor TCP one). I'm fine with closing this one - we can open a new one if this happens again. - thanks for debugging. - it seems that the splitting of the test actually made things waay better. Which is great.


Question: Interact with the kubernetes-mesos framework via `kubectl`: ```bash <del> $ bin/kubectl get pods POD        IP        CONTAINER(S)        IMAGE(S)        HOST        LABELS        STATUS </del> <ins> $ kubectl get pods NAME      READY     REASON    RESTARTS   AGE </ins> ``` ```bash <del> $ bin/kubectl get services       # your service IPs will likely differ NAME            LABELS                                    SELECTOR            IP             PORT kubernetes      component=apiserver,provider=kubernetes   <none>              10.10.10.2     443 ``` Lastly, use the Mesos CLI tool to validate the Kubernetes scheduler framework has been registered and running: ```bash $ mesos state | grep "Kubernetes" "name": "Kubernetes", </del> <ins> # NOTE: your service IPs will likely differ $ kubectl get services NAME             LABELS                                    SELECTOR   IP(S)          PORT(S) k8sm-scheduler   component=scheduler,provider=k8sm         <none>     10.10.10.113   10251/TCP kubernetes       component=apiserver,provider=kubernetes   <none>     10.10.10.1     443/TCP </ins> ``` <del> Or, look for Kubernetes in the Mesos web GUI by pointing your browser to `http://${mesos_master}`. Make sure you have an active VPN connection. </del> <ins> Lastly, look for Kubernetes in the Mesos web GUI by pointing your browser to `http://<mesos_master_ip:port>`. Make sure you have an active VPN connection. </ins> Go to the Frameworks tab, and look for an active framework named "Kubernetes". ## Spin up a pod

Hi We are updating docs to v1beta3. v1beta1 and v1beta2 will be abandoned soon. For the examples in getting-started-guides, we tend to use hyperlinks to point to examples instead of letting them have their own individual examples (see ). However, in the mesos guide, the examples contain mesos specific setup and knowledge, so I can't simply remove the texts and point to kubernetes/examples/guestbook, which is updated to v1beta3. My suggestion is that we update the guestbook example in kubernetes-mesos, and use a hyperlink in mesos getting-started-guide to replace the example texts. Could you help to make the changes? Thanks.
cc
We're going to delete v1beta1 and v1beta2 in O(days). See PR .
cc
we've initiated an effort to upstream some components into k8s. as part of that effort, we're upgrading the guestbook examples for k8s-mesos: the getting started guide for mesos will likely have to change in order to align with the move upstream. tracking this issue here: , Brian Grant wrote:
xref , James DeFelice wrote:
Ref
cc
xref
This needs to be moved to the v1 API.
Agreed. We have changes in review right now, should be merged soon. , Brian Grant wrote:


Question: github.com/google/go-cmp v0.4.0 github.com/gophercloud/gophercloud v0.1.0 github.com/mitchellh/mapstructure v1.1.2 <del> github.com/rubiojr/go-vhd v0.0.0-20160810183302-0bfd3b39853c </del> <ins> github.com/rubiojr/go-vhd v0.0.0-20200706105327-02e210299021 </ins> github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/testify v1.4.0 github.com/vmware/govmomi v0.20.3

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: envVarFiles := os.Getenv(RecommendedConfigPathEnvVar) if len(envVarFiles) != 0 { <del> chain = append(chain, filepath.SplitList(envVarFiles)...) </del> <ins> fileList := filepath.SplitList(envVarFiles) // prevent the same path load multiple times chain = append(chain, deduplicate(fileList)...) </ins> } else { chain = append(chain, RecommendedHomeFile)

/kind bug /sig cli What happened: The same path was multiple times to KUBECONFIG env variable. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-05T00:40:04Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"} Cloud provider or hardware configuration**: Irrelevant OS (e.g. from /etc/os-release): MacOS Sierra 10.12.6 Kernel (e.g. ): Darwin Kernel Version 16.7.0 Install tools: brew install kubernetes-cli Others:
I would help fix it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: structuralSchemas[v.Name] = s } <del> var openAPIModels proto.Models if utilfeature.DefaultFeatureGate.Enabled(features.ServerSideApply) && r.staticOpenAPISpec != nil { specs := []*spec.Swagger{} for _, v := range crd.Spec.Versions { s, err := builder.BuildSwagger(crd, v.Name, builder.Options{V2: false, StripDefaults: true, StripValueValidation: true}) if err != nil { utilruntime.HandleError(err) return nil, fmt.Errorf("the server could not properly serve the CR schema") } specs = append(specs, s) } mergedOpenAPI, err := builder.MergeSpecs(r.staticOpenAPISpec, specs...) if err != nil { utilruntime.HandleError(err) return nil, fmt.Errorf("the server could not properly merge the CR schema") } openAPIModels, err = utilopenapi.ToProtoModels(mergedOpenAPI) if err != nil { utilruntime.HandleError(err) return nil, fmt.Errorf("the server could not properly serve the CR schema") } </del> <ins> openAPIModels, err := buildOpenAPIModelsForApply(r.staticOpenAPISpec, crd) if err != nil { utilruntime.HandleError(fmt.Errorf("error building openapi models for %s: %v", crd.Name, err)) openAPIModels = nil </ins> } for _, v := range crd.Spec.Versions {

What happened: While upgrading from v1.15 to try out kubernetes v1.16.0-beta.2, kube-apiserver is printing error messages: This also brakes the API server since the user is unable to even delete the "faulty" CR What you expected to happen: Not to print any error message and accept a CR that was already valid in 1.15 How to reproduce it (as minimally and precisely as possible): Install k8s 1.15.x (optional) Create the following CR Upgrade to k8s v1.16.0-beta.2check kube-apiserver for errors:Try delete the faulty CR (Won't be possible) Environment: Kubernetes version (use ): Anything else we need to know?: A quick search sent me to this code path that is being used to compile k8s It seems the problem it known but not yet fixed since this is in the latest commit of : /kind bug /king regression cc
/sig api-machinery
/assign /priority critical-urgent /milestone v1.16
This is caused by compiling the openapi spec for use by server-side apply here: SSA is using the method, which has significant gaps in handling valid openapi specs (see all the TODO comments in ) and until now was only used for client-side kubectl validation (as far as I'm aware). Using it in the server-side path is concerning. /assign
running with , this works as expected and is compatible with 1.15
/assign
fix and covering test is in
this is resolved in , available now


Question: }, }, }, <ins> { predicates:   map[string]algorithm.FitPredicate{"true": truePredicate}, prioritizers: []algorithm.PriorityConfig{{Map: falseMapPriority, Weight: 1}, {Map: trueMapPriority, Reduce: getNodeReducePriority, Weight: 2}}, nodes:        []string{"2", "1"}, pod:          &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "2"}}, name:         "test error with priority map", wErr:         errors.NewAggregate([]error{errPrioritize, errPrioritize}), }, </ins> } for _, test := range tests { t.Run(test.name, func(t *testing.T) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: live := &unstructured.Unstructured{} live.SetKind(gvk.Kind) live.SetAPIVersion(gvk.GroupVersion().String()) <del> f = NewStripMetaManager(f) f = NewManagedFieldsUpdater(f) f = NewBuildManagerInfoManager(f, gvk.GroupVersion()) f = NewProbabilisticSkipNonAppliedManager(f, &fakeObjectCreater{gvk: gvk}, gvk, DefaultTrackOnCreateProbability) f = NewLastAppliedManager(f, typeConverter, objectConverter, gvk.GroupVersion()) f = NewLastAppliedUpdater(f) </del> <ins> f = NewLastAppliedUpdater( NewLastAppliedManager( NewProbabilisticSkipNonAppliedManager( NewBuildManagerInfoManager( NewManagedFieldsUpdater( NewStripMetaManager(f), ), gvk.GroupVersion(), ), &fakeObjectCreater{gvk: gvk}, gvk, DefaultTrackOnCreateProbability, ), typeConverter, objectConverter, gvk.GroupVersion(), ), ) </ins> if chainFieldManager != nil { f = chainFieldManager(f) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-integration runkubectldiff_tests It only shows failure cases in recent 2 days. The last date is I tried to reproduce it locally, but can't
/sig testing
/sig cli
/cc
I’ve only briefly taken a look at this, but my understanding is that server-side diff is flaking because the timestamp is different (by like a second) between the one already applied and the diff apply. do you think this could be related to this recent change Could delaying when we update the timestamp cause it to sometimes get misaligned with what’s being diffed? I’m not super familiar with diff, but it feels like it should ignore differences between timestamps that get updated on each call, right? Is there some sort of mechanism we have in that knows to ignore timestamps that just isn’t being done for fields?
This error probably shows that we have a path that causes the object to be re-written to etcd when it shouldn't.
As we can see, the diff only shows a difference in the timestamp but not in the fields that are managed. We are not supposed to update the timestamp if the managed fields haven't changed. That timestamp only gets updated if the two steps happen in different seconds (that's why it's flaky). That is definitely not a flake but an actual failure.
Failed 81/236 times today in /priority critical-urgent I think, correct it if I am wrong.
/triage accepted
correct. who should be assigned as the owner of this?
I can take it, I'm looking into it now
/assign


Question: // pods. func (kl *Kubelet) GetPods() []*v1.Pod { pods := kl.podManager.GetPods() <del> // a kubelet running without apiserver requires an additional // update of the static pod status. See #57106 </del> for i, p := range pods { <ins> // Pod cache does not get updated status for static pods. // TODO(tallclair): Most callers of GetPods() do not need pod status. We should either parameterize this, // or move the status injection to only the callers that do need it (maybe just the /pods http handler?). </ins> if kubelettypes.IsStaticPod(p) { if status, ok := kl.statusManager.GetPodStatus(p.UID); ok { <del> klog.V(2).InfoS("Pod status updated", "pod", klog.KObj(p), "status", status.Phase) </del> // do not mutate the cache p = p.DeepCopy() p.Status = status

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I would like to ask to increase the log level for this line: Why is this needed: I do not see the necessity to show every time the static pod has the status updated. It can flood the logs with the message. I do not know if there is a reason for that. I believe it is a matter of taste. Furthermore, the adding this piece of code, the log was not there. The log line has being in a
/sig node
I'll be happy to send a merge request for this issue. But I just need to know if this make sense to the community or not.
According to my quick research, this log is in during fixing a regression issue. Agree with you. I think it makes sense to use a lower log level as the bug is already fixed. However, I'm not the owner of kubelet. Let node/kubelet owner make the decision.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
is the issue that it logs the same status over and over? Or that status being changed very often?
/triage accepted I agree V(2) seems a bit high for this. Feel free to submit a PR to drop to 3.
what is the reason for rapid status change?
It the almost the same message. This is an example: I'm willing to send the PR.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign This is still an issue.
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten As PR is still open


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> TAG = 0.5 </del> <ins> TAG = 0.6 </ins> PREFIX = gcr.io/google_containers all: push

GCE minion VM OS version is The nfs server image and hits with the following dump stack
Image version now exists!
thanks!


Question: return nil } <del> // translateAllowedTopologies translates allowed topologies within storage class </del> <ins> // translateAllowedTopologies translates allowed topologies within storage class or PV </ins> // from legacy failure domain to given CSI topology key func translateAllowedTopologies(terms []v1.TopologySelectorTerm, key string) ([]v1.TopologySelectorTerm, error) { if terms == nil {

Unknown allowedTopology labels cause csi translation to fail: However, on GKE other labels are sometimes used in PV node affinity, such as or It's not common (100s of clusters as of July 2021), but seems enough that it should be dealt with. The suggestion is to just pass unknown topologies through. If they are used in the storageclass labels they will cause failures, but that is the current gce-pd behavior anyway. This reasoning probably holds for other clouds as well, although I haven't tested it myself. /assign /sig storage
/triage accepted


Question: moreForeverTestTimeout = 2 * 60 * time.Second ) <del> var _ = SIGDescribe("DNS configMap federations", func() { </del> <ins> var _ = SIGDescribe("DNS configMap federations [Feature:Federation]", func() { </ins> t := &dnsFederationsConfigMapTest{dnsTestCommon: newDnsTestCommon()}

/kind failing-test /priority critical-urgent /sig network /milestone v1.12 /cc
ping
Also failures on
TLS errors connection refused 503 service unavailable especially around metrics and healthz in ways that make me suspect things didn't start correctly. Or perhaps API mismatch? /cc 've looked at some similar failures in the past. Any chance you might give a look over this and ?
in both and here looks like api mismatch?
Test output shows no DNS server can be reached, but logs say coredns pods are running (https://k8s-): cc
The CoreDNS configmap is incorrectly configured and hence the test fails. I'm working on the fix.
/assign
, , Does it make sense to keep that e2e test ?, knowing that this test is specific for "Federation" and that federation has now merged into "multicluster" and removed all e2e tests about "Federation + Serial" (formerly "ci-federation-e2e-gce-serial") : can you say if that use case is still supported in kubernetes v1.12 ?
I can say that Federation v1 has not been exercised since v1.10. I think a good question for the Federation v2 folks is not only whether they still need the tests, but also whether they need the configmap support specific to what Federation v1 used -- I'll defer to
If ya'll decide the use case is no longer valid, please open an issue / remove the test once 1.13 release opens.
tests are failing
/reopen /cc
Reopening this issue. <detailsIn response to : /reopen /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
More failures:
Looks like this is failing since Kube-dns is now enabled.
Since we'd previously discussed this not being a relevant test case, and it seems less appropriate even if it's this coupled to the dns implementation and we've got the kube-dns to coredns transition happening in the next release most likely....Can we just disable/remove the test?
Opened to disable this test.
Thank you!


Question: if _, err := exec.LookPath("git"); err != nil { log.Fatalf("required git executable not found: %v", err) } <del> if err := syncRepo(*flRepo, *flDest, *flBranch, *flRev); err != nil { log.Fatalf("error syncing repo: %v", err) </del> <ins> for { if err := syncRepo(*flRepo, *flDest, *flBranch, *flRev); err != nil { log.Fatalf("error syncing repo: %v", err) } log.Printf("wait %d seconds", *flWait) time.Sleep(time.Duration(*flWait) * time.Second) log.Println("done") </ins> } <del> log.Printf("wait %d seconds", *flWait) time.Sleep(time.Duration(*flWait) * time.Second) log.Println("done") </del> } // syncRepo syncs the branch of a given repository to the destination at the given rev.

This causes . In contrib/git-sync demo, the container voluntarily exits and gets recreated to "periodically" sync the git repo, but the pod is then reported as not ready, The end result is that the service targets at this pod does not get endpoints and thus not working. Thanks to for helping with the debugging.
The container was not running when kubelet reported the status, hence it was considered not ready. Because the containers in a pod share the same fate, the pod was not perceived as not ready. Having short-lived containers as a service endpoint may encounter problems like this. I am not really convinced that this is kubelet's problem. It sounds like should be a cron job...
Forgot to cc
Is it alright if I modify git-sync to make it run a for-loop?
sure, but it would still be nice to able to have a way to have background containers restarting periodically in a pod w/o failing the service.
, will it be useful if we let the user to config which containers will be considered by kubelet when it reports the readiness of the pod?
I am not sure kubelet restart is the right mechanism to run periodic tasks, and in this case, recreating the container isn't particularly efficient. FYI, there is a discussion for adding a controller for cron jobs in .
Thanks Let's forget about letting kubelet restart periodic tasks. In general, if some containers are not critical to a service, ignoring the status of these containers and keeping the service alive will increase the availability of the service.
we'd like to have sidecar containers that aren't critical to a pod's overall functioning. i.e. a sidecar which can crash or become unavailable without pulling the pod from the load balancer. In our case it is a sidecar which delivers some configuration to disk from a notoriously flakey service. We'd like the pod to stay up, the conf it needs is on disk. Should I file a separate issue for this?


Question: // Wait till grpc server is ready. for i := 0; i < 10; i++ { services := m.server.GetServiceInfo() <del> if len(services) > 1 { </del> <ins> if len(services) > 0 { </ins> break } time.Sleep(1 * time.Second)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For now, UT in doesn't work because callback function will never be called, is rewritten in this function : Another thing, i'm wondering why we rewrite device health in function which in this file: What you expected to happen: UT assertion should be executed. How to reproduce it (as minimally and precisely as possible): Add some logs in callback function of , it will show nothing. Anything else we need to know?: I will send a PR to fix this. /cc WDYT? /sig node
/assign
GitHub didn't allow me to assign the following users: RenaudWasTaken. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Good catch


Question: importpath = "k8s.io/kubectl/pkg/cmd/create", visibility = ["//build/visible_to:pkg_kubectl_cmd_create_CONSUMERS"], deps = [ <del> "//staging/src/k8s.io/api/apps/v1:go_default_library", </del> "//staging/src/k8s.io/api/batch/v1:go_default_library", "//staging/src/k8s.io/api/batch/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: if !mutationDetectionEnabled { return dummyMutationDetector{} } <ins> glog.Warningln("Mutation detector is enabled, this will result in memory leakage.") </ins> return &defaultCacheMutationDetector{name: name, period: 1 * time.Second} }

/kind bug What happened: Start a cluster using ./hack/local- Now just monitor the memory on kube-apiserver and kube-controller-manager. Without doing anything else note the memory on each process has grown by 500 MB each day. Tried this for 5 days and saw this steady growth. What you expected to happen: Memory to remain relatively flat after some initial growth. How to reproduce it (as minimally and precisely as possible): Run ./hack/local- and wait. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10 Cloud provider or hardware configuration: local OS (e.g. from /etc/os-release): Debian GNU/Linux Kernel (e.g. ): Debian 4.9.65-3+deb9u2 Install tools: Others:
/sig api-machinery
/milestone v1.10
/priority important-soon
!
Copying some of what was discussed in Slack - we need to redo a test with set when running . This disables the cache mutation detector, which holds on to "all the things" .
What does the chart mean? yeah determined that most of it is from the mutation detector (it really makes me nervous to have test code linked in to the production binaries).
It was a differential pprof memory chart of hack/local-cluster- But as noted, we have to run it without mutation detection. I did that for an hour yesterday and did not notice much memory leaking.
After the suggested fix (turn of mutation detection), there was no noticeable increase on controller manager memory and the api-server memory increase was down to 8 MB per day. (on my setup)
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 2 days, the issue will be moved out of the v1.10 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
closing this to remove it from the release radar. Feel free to reopen if you think we still have an issue.


Question: <ins> apiVersion: v1 kind: ConfigMap metadata: name: test </ins>

What happened: I get a diff, when I specify a file directly, but on recursive mode, no diff is shown anymore: What you expected to happen: Recursive mode should work How to reproduce it (as minimally and precisely as possible): Have a bunch of YAML manifest, but with name collisions, e.g. Service , Deployment , etc Anything else we need to know?: I think the problem is that all objects have the same name, which should not be a problem per se. Only heapster-binding has a different name, and looking at the generated temp files, only and are present: Environment: Kubernetes version (use ): $ kubectl version --short Client Version: v1.13.0 Server Version: v1.13.0 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
I think the files are overwriting each other?
Yeah, good catch, I'll try to fix that asap
Thanks! By the way, very nice work with dry-run and diff. I was waiting for this.
Feel free to look at the PR and give your opinion!


Question: }, }, failedNodeToStatusMap: framework.NodeToStatusMap{ <del> "node-a": framework.NewStatus(framework.Unschedulable, algorithmpredicates.ErrTopologySpreadConstraintsNotMatch.GetReason()), "node-b": framework.NewStatus(framework.Unschedulable, algorithmpredicates.ErrTopologySpreadConstraintsNotMatch.GetReason()), "node-x": framework.NewStatus(framework.Unschedulable, algorithmpredicates.ErrTopologySpreadConstraintsNotMatch.GetReason()), </del> <ins> "node-a": framework.NewStatus(framework.Unschedulable, podtopologyspread.ErrReasonConstraintsNotMatch), "node-b": framework.NewStatus(framework.Unschedulable, podtopologyspread.ErrReasonConstraintsNotMatch), "node-x": framework.NewStatus(framework.Unschedulable, podtopologyspread.ErrReasonConstraintsNotMatch), </ins> }, nodeNames: []string{"node-a/zone1", "node-b/zone1", "node-x/zone2"}, registerPlugin: st.RegisterPluginAsExtensions(

Part of /priority important-soon See example /sig scheduling /kind cleanup
/help
/assign
/remove-help