Question: } } <del> // NewNoPathDefined creates a new MetricsError with code NoPathDefined. </del> <ins> // NewNotSupportedErrorWithDriverName creates a new MetricsError with code NotSupported. // driver name is added to the error message. func NewNotSupportedErrorWithDriverName(name string) *MetricsError { return &MetricsError{ Code: ErrCodeNotSupported, Msg:  fmt.Sprintf("metrics are not supported for %s volumes", name), } } // NewNoPathDefinedError creates a new MetricsError with code NoPathDefined. </ins> func NewNoPathDefinedError() *MetricsError { return &MetricsError{ Code: ErrCodeNoPathDefined,

What happened: kubelet crashed due to nil pointer dereference inside calcAndStoreStats function in Jul 03 06:50:42 kubelet[4440]: E0703 06:50:42. 4440 ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Jul 03 06:50:42 kubelet[4440]: goroutine 3224 [running]: Jul 03 06:50:42 kubelet[4440]: (0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x7b Jul 03 06:50:42 kubelet[4440]: (0x0, 0x0, 0x0) Jul 03 06:50:42 kubelet[4440]: +0x82 Jul 03 06:50:42 kubelet[4440]: panic(0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x1b5 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).parsePodVolumeStats(0xc00178ef00, 0xc000c96cc0, 0xe, 0xc0014b8880, 0x0, 0xc000c96cc0, 0xe, 0x0, 0x0, 0x0, .) Jul 03 06:50:42 kubelet[4440]: +0x73 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).calcAndStoreStats(0xc00178ef00) Jul 03 06:50:42 kubelet[4440]: +0x753 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).StartOnce.func1.

1() Jul 03 06:50:42 kubelet[4440]: +0x2a Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10) Jul 03 06:50:42 kubelet[4440]: +0x54 Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10, 0xdf8475800, 0x3ff0000000000000, 0xc000f1dd01, 0xc0012c37a0) Jul 03 06:50:42 kubelet[4440]: +0xf8 Jul 03 06:50:42 kubelet[4440]: created by (volumeStatCalculator).StartOnce.func1 Jul 03 06:50:42 kubelet[4440]: +0x9d I think this happens because GetMetrics function can return nil for both metric(), and error, but calcAndStoreStats expects error to be not nil when metric is nil() What you expected to happen: Skip calculating stats for volumes that do not support volume stats. How to reproduce it (as minimally and precisely as possible): Create a volume using a csi driver that does not support volume stats. Anything else we need to know?: Environment: Kubernetes version (use ): Server version 15.0 Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 16.04 LTS Kernel (e.g. ): 4.4.0-141-generic Install tools:kubeadm Network plugin and version (if this is a network-related bug): Others: Tested using synology-csi driver()
Sounds like you have a PR ready. Mind letting us know the PR number ? Thanks
I do not have a PR yet, but happy to make one. I'll add the PR number once I created it.
I created a PR and linked to this issue.
Didnt this solve this issue ?
No, it does not fix the problem that metric itself can be nil. Also, my pull request fixes the problem that csi_metrics returns nil value for error and the caller expects non nill value when csi driver does not support metrics.


Question: imageutils "k8s.io/kubernetes/test/utils/image" ) <ins> const ( rsRetryTimeout = 2 * time.Minute ) </ins> func newRS(rsName string, replicas int32, rsPodLabels map[string]string, imageName string, image string, args []string) *appsv1.ReplicaSet { zero := int64(0) return &appsv1.ReplicaSet{

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } for vmssName := range vmssNamesMap { <del> vmss, err := ss.GetScaleSetWithRetry(service, ss.ResourceGroup, vmssName) </del> <ins> vmss, err := ss.getVMSS(vmssName, cacheReadTypeDefault) </ins> if err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleUpIgnoresFailedPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  2, expectedReplicas: 4, podReadiness:     []v1.ConditionStatus{v1.ConditionTrue, v1.ConditionTrue, v1.ConditionFalse, v1.ConditionFalse}, podPhase:         []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodFailed, v1.PodFailed}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, levels:   []int64{500, 700}, targetUtilization:   30, expectedUtilization: 60, expectedValue:       numContainersPerPod * 600, }, } tc.runTest(t) } </ins> func TestReplicaCalcScaleUpCM(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3,

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: other platforms like Windows. release: v1.9 file: test/e2e/common/network/networking.go <ins> - testname: Proxy, validate Proxy responses codename: '[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service Proxy [Conformance]' description: Attempt to create a pod and a service. A set of pod and service endpoints MUST be accessed via Proxy using a list of http methods. A valid response MUST be returned for each endpoint. release: v1.24 file: test/e2e/network/proxy.go </ins> - testname: Proxy, validate ProxyWithPath responses codename: '[sig-network] Proxy version v1 A set of valid responses are returned for both pod and service ProxyWithPath [Conformance]'

needs to be resolved before a number of e2e tests can be completed. 23 Oct 2021 [x] two weeks soak end date : 5 Nov 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining PodProxyOptions endpoints which are untested. As the these endpoints are redirected to their related endpoints. Create pod and confirm that the pod is running so that the tested endpoints can be found. Create a http.Client that checks for a redirect so that status code can be checked Loop through all http verbs, testing that the pod proxy endpoint returns the required 301 status code. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. After the merger of the test was updated to cover both the Pod and Service Proxy endpoint. Therefore If a test with these calls gets merged, test coverage will go up by 12 points connectCoreV1DeleteNamespacedPodProxy connectCoreV1DeleteNamespacedServiceProxy connectCoreV1OptionsNamespacedPodProxy connectCoreV1OptionsNamespacedServiceProxy connectCoreV1PatchNamespacedPodProxy connectCoreV1PatchNamespacedServiceProxy connectCoreV1PostNamespacedPodProxy connectCoreV1PostNamespacedServiceProxy connectCoreV1PutNamespacedPodProxy connectCoreV1PutNamespacedServiceProxy connectCoreV1HeadNamespacedPodProxy onnectCoreV1HeadNamespacedServiceProxy /sig testing /sig architecture /area conformance
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale To be revisited in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale Focus for 1.22
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Focus for 1.23
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale Focus for 1.23 Will move forward once merged
/open Waiting for 2 week flake free before promotion
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


What happened: Created a CRD: Created another CRD defining an overlapping type with CustomResourceDefinition: Verify both exist: Deleted the one overlapping with the in-tree CRD type: Verify only the overlapping CRD remains: and is stuck deleting: What you expected to happen: Creation and deletion of the overlapping type CRD would have no effect on the in-tree data. Anything else we need to know?: The only types in the kube-apiserver (currently) that store data under a group prefix are: APIService () CustomResourceDefinition () Custom resources () When a CRD is deleted, all instances of the type it defines are first removed. That means that defining and deleting a CustomResourceDefinition that defines an overlapping APIService or CustomResourceDefinition type will remove existing APIService or CustomResourceDefinition data. We can special-case these two resources in the CRD deletion code, but we need a more robust solution before CRD v1. /cc /sig api-machinery /priority important-soon /area custom-resources
/sub
Is anyone working on this? I'll like to take a stab at this.
Not currently, no. Great. I'd suggest discussion a potential solution here before diving into code. The two main aspects are: -time - Should we forbid creating CRDs that overlap with built-in types? -time - How do we detect a CRD overlapping with a built-in type and skip the etcd cleanup for it
I think we should forbid creating CRDs that overlap (option 1).
That can help, but is not sufficient. Imagine: n forbids creating overlapping CRDs on version n creates MyType, which is allowed because that is not a built-in type upgrades to version n+1 which adds MyType as a built-in type deletes CRD What should happen?
For that, we may need delete-time detection of overlapping types as well (option 2), which may make option 1 redundant. Do you have any advice for detecting overlapping CRDs?
for now, just delete-time detection in


Question: <ins> [SaltStack reactor](http://docs.saltstack.com/en/latest/topics/reactor/) files, largely defining reactions to new nodes. **Ignored for GCE, which runs standalone on each machine** </ins>

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: servicePorts = append(servicePorts, int(servicePort.NodePort)) } } <ins> if service.Spec.HealthCheckNodePort != 0 { servicePorts = append(servicePorts, int(service.Spec.HealthCheckNodePort)) } </ins> return servicePorts }

I came across a cluster where it has 2 services with port collision. One service looks like this: The other looks like: I checked the logic that allocates nodeport and healthcheck nodeport. They went thru the same node port allocator. Port allocation mainly happens in , and . The file in question is
cc
/cc
I'm not clear on your context. Which case were you facing? config and create the two services as you mentioned manifests, then get errors due to the collided ports? create the two services without specifying the two ports, then get the services and find they have collided ports? Also, what is the expected behavior you were thinking of?
When I checked the cluster, there were 2 services with colliding ports. I am not sure what brought the cluster into this state.
It's strange... I will try to dig into it later.
Seems like there is a race condition. PR sent. PTAL.
I first thought there was a race condition too. But I checked the API log. The operation on the 2 services are quite separated. Then I am not sure.
Oh... So strange. I will take another look at the code to try to find some clues. Btw, does the lock to make any sense? It doesn't fix this issue though.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
you can't re-open an issue/PR unless you authored it or you are assigned to it.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
/assign
Found the issue. It's due to the nodeport repair controller ignoring the healthCheckNodePort.
Will send a PR and cherrypicks.


Question: Volumes       []Volume      `json:"volumes"` Containers    []Container   `json:"containers"` RestartPolicy RestartPolicy `json:"restartPolicy,omitempty"` <del> // Optional: Set DNS policy.  Defaults to "ClusterFirst" </del> <ins> // Required: Set DNS policy. </ins> DNSPolicy DNSPolicy `json:"dnsPolicy"` }

Many comments in there say "if not specified, default is ..." - but we previously asserted that there are no defaults for the internal API. We should strip those comments out, and leave them in versioned structs only.
cc


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "time" <ins> autoscalingv1 "k8s.io/api/autoscaling/v1" </ins> v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/pkg/controller/replication" "k8s.io/kubernetes/test/e2e/framework"

According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. Create a ReplicationController with a static label Patch the ReplicationController with a new Label and updated data Patch the ReplicationController's status Patch/Scale the ReplicationController's Replica count to 2 Replicas Get the ReplicationController Replace the ReplicationController's Status List all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Delete Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Deployment replaced ReplicationController a while back. It should be part of conformance. We need to check more than the data-aspect (beyond crud), we need to verify the data-plane. In this case we need to check the pods as well and not just the ReplicationController status. Moved to backlog by conformance subgroup.
PR:
/open /reopen We need this open to track the test promotion and flakyness. We can't write a promotion PR until the test has no flakes for 2 weeks.
Reopened this issue. <detailsIn response to : /open /reopen We can't write a promotion PR until the test has no flakes for 2 weeks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1ReplicationController lifecycle mock test - +7 coverage Need to verify two weeks on non-flakeyness.
/retitle Write CoreV1ReplicationController lifecycle test+promote - +7 coverage
Closing in favour of /close
Closing this issue. <detailsIn response to : Closing in favour of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectWarning              string expectFatal                bool expectDelete               bool <ins> expectOutputToContain      string </ins> }{ { <del> description:  "RC-managed pod", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{rcPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "RC-managed pod", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{rcPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { description:  "DS-managed pod",

When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted


Question: // k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/egressselector "apiserver_egress_dialer_dial_failure_count", // counter metrics should have "_total" suffix <del> // k8s.io/apiserver/pkg/util/flowcontrol/fairqueuing/queueset "apiserver_flowcontrol_current_inqueue_requests",   // label names should be written in 'snake_case' not 'camelCase', "apiserver_flowcontrol_current_executing_requests", // label names should be written in 'snake_case' not 'camelCase' "apiserver_flowcontrol_rejected_requests_total",    // label names should be written in 'snake_case' not 'camelCase' </del> // k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/healthz "apiserver_request_total", // label names should be written in 'snake_case' not 'camelCase'

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: func RestartControllerManager() error { // TODO: Make it work for all providers and distros. if !ProviderIs("gce", "aws") { <del> return fmt.Errorf("unsupported provider: %s", TestContext.Provider) </del> <ins> return fmt.Errorf("unsupported provider for RestartControllerManager: %s", TestContext.Provider) </ins> } if ProviderIs("gce") && !MasterOSDistroIs("gci") { return fmt.Errorf("unsupported master OS distro: %s", TestContext.MasterOSDistro)

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by


Question: source "${KUBE_HOME}/kube-env" if [[ -f "${KUBE_HOME}/kubelet-config.yaml" ]]; then echo "Found Kubelet config file at ${KUBE_HOME}/kubelet-config.yaml" KUBELET_CONFIG_FILE_ARG="--config ${KUBE_HOME}/kubelet-config.yaml"

What happened: Pick an OS image where python is python3 and the cluster will not come up. What you expected to happen: Cluster should come up. How to reproduce it (as minimally and precisely as possible): Switch the default OS image to use an OS which has python3 as the default python. Run cluster/kube- for GCE. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/assign


Question: // reject to forward redirect response rejectForwardingRedirects bool <ins> // tracerProvider is used to wrap the proxy transport and handler with tracing tracerProvider tracing.TracerProvider </ins> } type proxyHandlingInfo struct {

Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted


Question: local REGION=${ZONE%-*} gcloud compute forwarding-rules delete -q --region ${REGION} "${INSTANCE_PREFIX}-default-frontend" || true gcloud compute target-pools delete -q --region ${REGION} "${INSTANCE_PREFIX}-default-frontend" || true <ins> gcloud compute firewall-rules delete guestbook-e2e-minion-8000 -q || true fi } function wait_for_running() { echo "Waiting for pods to come up." local frontends master slaves pods all_running status i pod frontends=($(${KUBECTL} get pods -l name=frontend -o template '--template={{range.items}}{{.id}} {{end}}')) master=($(${KUBECTL} get pods -l name=redis-master -o template '--template={{range.items}}{{.id}} {{end}}')) slaves=($(${KUBECTL} get pods -l name=redis-slave -o template '--template={{range.items}}{{.id}} {{end}}')) pods=("${frontends[@]}" "${master[@]}" "${slaves[@]}") all_running=0 for i in {1..30}; do all_running=1 for pod in "${pods[@]}"; do status=$(${KUBECTL} get pods "${pod}" -o template '--template={{.currentState.status}}') || true if [[ "$status" != "Running" ]]; then all_running=0 break fi done if [[ "${all_running}" == 1 ]]; then break fi sleep 10 done if [[ "${all_running}" == 0 ]]; then echo "Pods did not come up in time" return 1 </ins> fi }

Part of issue . Re-write test in Go, generate unique names for top level resources, ensure test is idempotent, clean up all resources at end of test.
What should we do about this test? The current test in only brings up the guestbook but doesn't even confirm that it is up or tries to connect to any of the ports... Was the point of that test case to test /? Does it make sense to rewrite this test in Go or do we already have more test coverage from e.g. ? If it makes sense to rewrite it in Go, what should we accomplish with this test?
How about a functional test that inserts some names and tries to retrieve them i.e. enough to know that the service has basic functionality?
+1 to an actual functional test. The service is basic enough, it should be straightforward to do the same POST/GET from go that the js on the guestbook homepage does.
and I talked about this and for a while and think that they should remain as tests. This test needs to be improved: it needs to be turned into an actual test (we don't do anything with the service).
has not passed on gce or gke-ci since around 11am this morning (around when this PR was merged). Can you look at the Jenkins logs and try to figure out what is going wrong?
The problem is with guestbook itself not with the test. The frontend cannot connect to redis data base so it returns error response to the user. I created issue for that
I have been 100% absorbed for weeks with my own e2e test so I've not paid much attention to this. Does the test connect to the service via an external load balancer?
Nope. It's using proxy. For example:
Is this a reasonable use of the proxy service? Today told me that the proxy service is fine for contacting cluster-level services (note the dash Brian) but for regular applications we should use some other mechanism (e.g. external load balancer). ?
My first idea was to use ELB but it requires to create also firewall-rules using. I'm not sure if we want to do it.
PR has been reverted by The test is passing for me locally. I'll updated . Is there any chance that using proxy instead of ELB can cause the problem?

Personally, I think -- for now -- using an ELB is fine -- by setting the use external load balancer field in the System specification. I am working on a PR to fix an issue relating to the use of application endpoints using the proxy server (to address issue -- which can bite you if one of the fields of a JSON response contains a field called ).
It sounds to me like there are 2 issues: should the guestbook example make the guestbook app accessible to users? should the test test correctness? I think these are 2 totally different things. The right answer to (1) is clearly external IPs. That's what we want to teach users to do. (2) likely doesn't have the same requirements. Conflating the 2 cases might not be helpful. However, if you want to do that, use the external load balancer, not the proxy.
Yes, I am torn about this too. Another thing to consider is that a test that uses an external load balancer is going to take longer because it takes much longer to create a service with an external load balancer (almost instant with ELB vs. perhaps up to a minute with a load balancer) and then there is a teardown step as well when you delete the service (which deletes the forwarding rules etc.). As a point of principle we should decide if it is an approved use of the proxy mechansim for use in tests in order to make them more lightweight vs. using an ELB.
So there is agreement to use ELB. What about firewall rule? Port 8000 should be opened, I can do it by using gcloud, however it's not provider agnostic operation. Is there any other option for that?
Sadly not: I suggest just using gcloud / making the API calls for the firewall and keep this as a GCE specific test. Which is indeed a shame.
Reopening since the PR was reverted in


Question: autoprovision.Register(plugins) exists.Register(plugins) noderestriction.Register(plugins) <del> label.Register(plugins) </del> <ins> label.Register(plugins) // DEPRECATED in favor of NewPersistentVolumeLabelController in CCM </ins> podnodeselector.Register(plugins) podpreset.Register(plugins) podtolerationrestriction.Register(plugins)

In , we a Persistent Volume Controller with existing code that runs in cloud-controller-manager In , we cleaned up dependencies for AWS/GCE for the new controller Next steps, [x] Deprecate the PersistentVolumeLabel admission controller [ ] Remove the PersistentVolumeLabel admission controller once GCE and AWS can run as external cloud controller
/sig cluster-lifecycle
cc
/reopen
Changed the timeline for this to v1.10, sounds good as an initial plan?
/area platform/gce /sig gcp
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/unassign
adding kind/cleanup to this issue so that it can stay in 1.10. Please update the kind if that is incorrect. kind cleanup
/remove-lifecycle stale /lifecycle frozen is working on running the cloud controller in gce, but it's not there yet, so we shouldn't remove the PV label admission controller yet.
and should we drop the v1.10 milestone then for this issue?
reminder: this feature will be automatically dropped from 1.10 at Code Freeze tonight.
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing labels required for the v1.10 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
PVL admission controller was set for deprecation in 1.8, in 1.11 we'll be turning it off by default, users can still enable it if they need it which seems to be the use for GCE.
^ I'm not 100% aware of the status of PVLs in AWS/GCE, this is just the deprecation plan we had in place a while back. If there are any known issues with doing this please let me know
As long as the admission controller is not removed from cluster/gce/config-, then GCE should continue to work.

/remove-sig cluster-lifecycle
/assign /triage accepted
is it time to try this again?
I don't think we're ready. There's no replacement for it in the cloud controllers. The initial attempt at replacement using initializers was abandoned. I think we should consider a replacement using an admission webhook. cc
Could I suggest we make this the primary topic in an upcoming meeting? Maybe April 22nd Cloud Provider Extraction WG meeting? Assuming and can make it?
Yes that sounds good.
What do we think the way forward looks like?
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/remove-sig api-machinery Since sig cloud provider has taken care of this one. :)
do you know what the latest status is on this?
i saw this go by the other day, please see (search for )
xref:
!
For the record, replacement webhook:
Is this done now?
/close yes! deleted in
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } else { os.MkdirAll(dir, 0700) // parameters suggested by https://azure.microsoft.com/en-us/documentation/articles/storage-how-to-use-files-linux/ <del> options := []string{fmt.Sprintf("vers=3.0,username=%s,password=%s,dir_mode=0700,file_mode=0700", accountName, accountKey)} </del> <ins> options := []string{fmt.Sprintf("username=%s,password=%s", accountName, accountKey)} </ins> if b.readOnly { options = append(options, "ro") } mountOptions = volume.JoinMountOptions(b.mountOptions, options) <ins> mountOptions = appendDefaultMountOptions(mountOptions) </ins> } err = b.mounter.Mount(source, dir, "cifs", mountOptions)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): azurefile azurefile 0777 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Microsoft Azure Container Service OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Linux 4.4.0-28-generic -Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x8664 x8664 x8664 GNU/Linux What happened: I attempted to set up an Azure File Volume using the built-in volume type azureFile. Everything works, except that the files and directories defaults to permission 0777, always. When I tried to manually mount within the container, I could customise dirmode and filemode, meaning that this change should be possible. Reading the source code confirms it - dirmode=0777 and filemode=0777 seem hardcoded. What you expected to happen: Being able to supply dirmode and filemode to the mount. e.g. How to reproduce it (as minimally and precisely as possible): N/A. This is a feature request.

Anything else do we need to know: N/A
This is a big requirement, using an azure file volume for postgresql for instance fails due to permissions as initdb tries to change the permissions to be less restrictive but is unable to.
Evaluating compatibility between a certain setup of a volume mount and an arbitrary piece of software would be the responsibility of the integrator, wouldn't it? It seems a big requirement for K8s itself to have to ensure PostgreSQL can do what it wants. An integrator who tries to use Azure File Storage for PostgreSQL should know better to mount with the correct permission.
The point is that you cannot mount with permissions, they're hard coded to 0777.
Ah. Yeah. Sorry for misunderstanding the comment above. Let's say it is an "important" requirement :)
/sig azure
/assign andyzhangx
I have a for this, use mount options in storage class, below is an example:
Has this change made it into any release yet? I can't seem to find it... Or was the issue resolved in some other way? I'm facing the issue you originally reported, i.e. using a pod volume to an azureFile, and it always mounting with filemode and dirmode of 0755... and I need it to be 0777. Is there a workaround? TIA.
azure file mountOptions feature is available from v1.8.5, here is the example :
Thanks -- I was still having trouble until I the storageClassName: "" to the claim... then realized I was hitting issue which I can avoid by setting storageClassName: "default" to both the pv and pvc, and now everything works. Thanks again!


Question: } func (t *HPAUpgradeTest) test() { <ins> const timeToWait = 15 * time.Minute </ins> t.rc.Resume() By(fmt.Sprintf("HPA scales to 1 replica: consume 10 millicores, target per pod 100 millicores, min pods 1.")) t.rc.ConsumeCPU(10) /* millicores */ By(fmt.Sprintf("HPA waits for 1 replica")) <del> t.rc.WaitForReplicas(1) </del> <ins> t.rc.WaitForReplicas(1, timeToWait) </ins> By(fmt.Sprintf("HPA scales to 3 replicas: consume 250 millicores, target per pod 100 millicores.")) t.rc.ConsumeCPU(250) /* millicores */ By(fmt.Sprintf("HPA waits for 3 replicas")) <del> t.rc.WaitForReplicas(3) </del> <ins> t.rc.WaitForReplicas(3, timeToWait) </ins> By(fmt.Sprintf("HPA scales to 5 replicas: consume 700 millicores, target per pod 100 millicores, max pods 5.")) t.rc.ConsumeCPU(700) /* millicores */ By(fmt.Sprintf("HPA waits for 5 replicas")) <del> t.rc.WaitForReplicas(5) </del> <ins> t.rc.WaitForReplicas(5, timeToWait) </ins> // We need to pause background goroutines as during upgrade master is unavailable and requests issued by them fail. t.rc.Pause()

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: startTime := metav1.NewTime(*tc.initialStatus.startTime) job.Status.StartTime = &startTime } <ins> job.Status.Conditions = append(job.Status.Conditions, tc.initialStatus.conditions...) </ins> } key, err := controller.KeyFunc(job)

When the FailureTarget=False condition is manually, then the Job terminates. The condition should be ignored by the Job controller. the like this: the following script to repro: Issue: the Job gets terminated with the status like this: No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign Thanks!
/sig apps


Question: if err != nil { // Skip detaching this volume if unable to update node status klog.ErrorS(err, "UpdateNodeStatusForNode failed while attempting to report volume as attached", "volume", attachedVolume) <ins> // Add volume back to ReportAsAttached if UpdateNodeStatusForNode call failed so that node status updater will add it back to VolumeAttached list. // It is needed here too because DetachVolume is not call actually and we keep the data consistency for every reconcile. rc.actualStateOfWorld.AddVolumeToReportAsAttached(attachedVolume.VolumeName, attachedVolume.NodeName) </ins> continue }

Recently, the cpu load of my kube-apiserver gets higher after I expanded more pods and nodes into the cluster. And I found that some volumes declared by pods could not be mounted after the restart of pods, even though the pods were successful scheduled. here's the kubectl describe information: After checking the logs of attachdetachcontroller, I found that in , return an error, thus the was skipped. That should be related to the high load on my kube-apiserver as it occasionally returns 500 http code (but this may not be the root cause). Here is the error message: Here comes the problem: In my opinion, the next loop of would retry and , since the state of actualStateOfWorld is different from the state of desiredStateOfWorld. but it didn't happened. By recording some additional logs, I notice that the pod was scheduled to the destination node by the scheduler right after above error message appeared, which cause executed. That would write the volume info back to . After that, the next loop of would skip the detach operation for that volume, since desiredStateOfWorld was 'equal to' actualStateOfWorld. The log snippet is as follows: Finally, attachdetachcontroller would not attach the volume neither, because the states of desiredStateOfWorld and actualStateOfWorld are the same. attachdetachcontroller mistakenly thought the volume was attached, but it' not the case. when fail, attachdetachcontroller should retry to detach the volume until the the detach operation completed. However, it was interrupted by the update of desiredStateOfWorld. a log and before a pod mounting pv Then you'll found log message "force skip detaching" would disappear after a few loops of No response <details</details<detailsN/A </detailsNo response <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig storage
If fails, that means the volume has not been detached and the volumeattachment is still on the nodeStatus, then kubelet should see that the volume still attaches to the node, so I don't understand why your pod failed to mount, it may not be caused by the controller. what's the error for kubelet?
! There are . The one before was failed, but another which is at the end of was successfully executed. Thus it cause the kubelet coundn't get the volume in .
kubelet log repeat the message below infinitely:
Okay, I know what you said, actually this problem is similar with the one which was fixed by , but it is not completely. Give another pr below to fix this.
Can we have the fix () back-ported to 1.21? Given that this completes the , the initial fix for which was , it would make sense to have this follow-up fix also be to 1.21. Looks like there is a that we could get this into. We have a 100+ clusters deployed in production that are running 1.21. We see this bug at least once a week and it's a headache to deal with. Upgrading to 1.22 or higher is a big undertaking for us.


Question: return nil } } <ins> // RemoveOutOfDiskCondition removes stale OutOfDisk condition // OutOfDisk condition has been removed from kubelet in 1.12 func RemoveOutOfDiskCondition() Setter { return func(node *v1.Node) error { var conditions []v1.NodeCondition for i := range node.Status.Conditions { if node.Status.Conditions[i].Type != v1.NodeOutOfDisk { conditions = append(conditions, node.Status.Conditions[i]) } } node.Status.Conditions = conditions return nil } } </ins>

What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3


Question: <ins> # This DaemonSet was originally referenced from # https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/daemonset.yaml # The Dockerfile and other source for this daemonset are in # https://github.com/GoogleCloudPlatform/cos-gpu-installer apiVersion: apps/v1 kind: DaemonSet metadata: name: nvidia-driver-installer namespace: kube-system labels: k8s-app: nvidia-driver-installer spec: selector: matchLabels: k8s-app: nvidia-driver-installer updateStrategy: type: RollingUpdate template: metadata: labels: name: nvidia-driver-installer k8s-app: nvidia-driver-installer spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: cloud.google.com/gke-accelerator operator: Exists tolerations: - operator: "Exists" hostNetwork: true hostPID: true volumes: - name: dev hostPath: path: /dev - name: vulkan-icd-mount hostPath: path: /home/kubernetes/bin/nvidia/vulkan/icd.d - name: nvidia-install-dir-host hostPath: path: /home/kubernetes/bin/nvidia - name: root-mount hostPath: path: / initContainers: - image: gcr.io/cos-cloud/cos-gpu-installer:v20200701 name: nvidia-driver-installer resources: requests: cpu: 0.15 securityContext: privileged: true env: - name: NVIDIA_INSTALL_DIR_HOST value: /home/kubernetes/bin/nvidia - name: NVIDIA_INSTALL_DIR_CONTAINER value: /usr/local/nvidia - name: VULKAN_ICD_DIR_HOST value: /home/kubernetes/bin/nvidia/vulkan/icd.d - name: VULKAN_ICD_DIR_CONTAINER value: /etc/vulkan/icd.d - name: ROOT_MOUNT_DIR value: /root volumeMounts: - name: nvidia-install-dir-host mountPath: /usr/local/nvidia - name: vulkan-icd-mount mountPath: /etc/vulkan/icd.d - name: dev mountPath: /dev - name: root-mount mountPath: /root containers: - image: "gcr.io/google-containers/pause:3.2" name: pause </ins>  No newline at end of file

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .


Question: } else { socket, err := proxier.portMapper.OpenLocalPort(&lp) if err != nil { <del> glog.Errorf("can't open %s, skipping this externalIP: %v", lp.String(), err) </del> <ins> msg := fmt.Sprintf("can't open %s, skipping this externalIP: %v", lp.String(), err) proxier.recorder.Eventf( &v1.ObjectReference{ Kind:      "Node", Name:      proxier.hostname, UID:       types.UID(proxier.hostname), Namespace: "", }, api.EventTypeWarning, err.Error(), msg) glog.Error(msg) </ins> continue } replacementPortsMap[lp] = socket

When kube-proxy fails to open a nodeport, it gives up setting up the rest of the iptables rules for the service. Leaving the service not usable. Cannot even use it with clusterIP. Kube-proxy only logs this error in . No useful information can be retrieved with kubectl. Need to record a event at least.
Ideally would the changes be removed and then reported back? Is there a preferred log or place to return the error?
the event maybe a better solution; as other port maybe success.
Need to surface the problem somewhere. An event seems okay
, there's and in ; should we reuse it or create new recorders for ?
Just reuse it. After all, it is within the same component. Are you going to submit a PR?
sure, my pleasure.
Great! Thanks! Let me know when the PR is ready.


Question: }, expectedElements: []string{"Terminating (lasts 10y)"}, }, <ins> { name:   "test16", plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name:                       "bar", GenerateName:               "test-GenerateName", UID:                        "test-UID", CreationTimestamp:          metav1.Time{Time: time.Now()}, DeletionTimestamp:          &metav1.Time{Time: time.Now()}, DeletionGracePeriodSeconds: new(int64), Labels:      map[string]string{"label1": "label1", "label2": "label2", "label3": "label3"}, Annotations: map[string]string{"annotation1": "annotation1", "annotation2": "annotation2", "annotation3": "annotation3"}, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ Local: &api.LocalVolumeSource{}, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{ NodeSelectorTerms: []api.NodeSelectorTerm{ { MatchExpressions: []api.NodeSelectorRequirement{ { Key:      "foo", Operator: "In", Values:   []string{"val1", "val2"}, }, { Key:      "foo", Operator: "Exists", }, }, }, }, }, }, }, }, expectedElements: []string{"Node Affinity", "Required Terms", "Term 0", "foo in [val1, val2]", "foo exists"}, }, { name:   "test17", plugin: "local", pv: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name:                       "bar", GenerateName:               "test-GenerateName", UID:                        "test-UID", CreationTimestamp:          metav1.Time{Time: time.Now()}, DeletionTimestamp:          &metav1.Time{Time: time.Now()}, DeletionGracePeriodSeconds: new(int64), Labels:      map[string]string{"label1": "label1", "label2": "label2", "label3": "label3"}, Annotations: map[string]string{"annotation1": "annotation1", "annotation2": "annotation2", "annotation3": "annotation3"}, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ CSI: &api.CSIPersistentVolumeSource{ Driver:       "drive", VolumeHandle: "handler", ReadOnly:     true, VolumeAttributes: map[string]string{ "Attribute1": "Value1", "Attribute2": "Value2", "Attribute3": "Value3", }, }, }, }, }, expectedElements: []string{"Driver", "VolumeHandle", "ReadOnly", "VolumeAttributes"}, }, </ins> } for _, test := range testCases {


Question: kube::util::go_install_from_commit github.com/kubernetes/repo-infra/kazel 4eaf9e671bbb549fb4ec292cf251f921d7ef80ac kube::util::go_install_from_commit github.com/bazelbuild/rules_go/go/tools/gazelle/gazelle 82483596ec203eb9c1849937636f4cbed83733eb <ins> touch "${KUBE_ROOT}/vendor/BUILD" </ins> gazelle fix -build_file_name=BUILD,BUILD.bazel -external=vendored -mode=fix -repo_root="$(kube::realpath ${KUBE_ROOT})" kazel -root="$(kube::realpath ${KUBE_ROOT})"

At this point I don't know if it is a valid file or not?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I reproduce it on my env. After removing , will not create and update ; Manually creating an empty file , will successfully rewrite and update ; /cc Seems file will not be created automatically if missing.
This is working as intended, or at least as designed. We only create new files if there are go sources in that directory, and doesn't have any go sources, so won't be auto-created. We have a solely for the and rules, but those are only managed in existing files. Without , we'd have a lot of lines in , which would be painful. I learned last week that the say to remove all of , which mucks with this process a little bit. I a step in the instructions there to recreate the file, but maybe should just be doing that? I'm not super excited about adding special cases there, but is kinda weird, so maybe it's unavoidable.
the more automatic the better
Specifically, this would cause many merge conflicts.
I'm a little confused, since this has been the way things have worked since merged in April, but 4+ people asking me about this in the last week have convinced me that this needs to be more automatic. Sent .


Question: testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestMaster(t, fmt.Sprintf("postfilter%v-", i), nil), <del> tt.numNodes, </del> <ins> int(tt.numNodes), </ins> scheduler.WithProfiles(prof), scheduler.WithFrameworkOutOfTreeRegistry(registry), )

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: integration-master Which test(s) are flaking: TestPostFilterPlugin Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /remove-sig testing
/assign


Question: }, "time": { SchemaProps: spec.SchemaProps{ <del> Description: "Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'", </del> <ins> Description: "Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over.", </ins> Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Time"), }, },

Godoc says that Time should be empty on mangedFields entry created by "Apply" operation, but kubernetes populates time on apply. Godoc consistent with kubernetes behaviour Godoc: Kubernetes: No response <details</details<details</details<details</details<details</details<details</details<details</details
/wg api-expression /sig api-machinery
/cc
/cc /triage accepted
Yeah, good catch. The comment matches the behavior from alpha, which we changed later on. Thanks for letting us now :-) I think we could do a better job describing how this value is changed now.


Question: then exit 1; fi; <del> LAST_MODIFIED_DATE=`stat /var/log/fluentd-buffers | grep Modify | sed -r "s/Modify: (.*)/1/"`; LAST_MODIFIED_TIMESTAMP=`date -d "$LAST_MODIFIED_DATE" +%s`; if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $STUCK_THRESHOLD_SECONDS` ]; </del> <ins> touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [[ -z "$(find /var/log/fluentd-buffers -type f -newer /tmp/marker-stuck -print -quit)" ]]; </ins> then rm -rf /var/log/fluentd-buffers; exit 1; fi; <del> if [ `date +%s` -gt `expr $LAST_MODIFIED_TIMESTAMP + $LIVENESS_THRESHOLD_SECONDS` ]; </del> <ins> touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [[ -z "$(find /var/log/fluentd-buffers -type f -newer /tmp/marker-liveness -print -quit)" ]]; </ins> then exit 1; fi;

This test is failing in gke serial suite: http://k8s- /sig node /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign also cc in case you already have some idea xref
/milestone v1.10
this also failed in upgrade jobs: http://k8s- xref
cc please update
/priority critical-urgent
Both are failing because the runtime cpu usage is exceeding our limit. In both cases, we are seeing 350m usage and expecting <200m usage
is this indicative of a general increase in resource usage, as per
I don't think so. This is a specific increase in the runtime (docker in this case) cpu usage
This seems like a legit issue. Docker's 95th% cpu usage has increased on all nodes since 2/27, when no test/user pods are running. The tests started failing when this PR landed: The PR bumped the fluentd version to v3.0, and I kept seeing fluentd liveness probes failures in the kubelet log. Not sure if it's related, but it's worth looking into.
^^^^^^^^ /cc
we should investigate this with according to this report, the latest version of fluentd using our image is using more CPU that is acceptable. I didn't see this behavior in my tests, I wonder if it has anything to do with the parser fixes we when we moved the multiformat into a plugin block.
Just ran my perf tests against the latest configuration, and I'm seeing 600m usage for CPU. Will investigate how we can optimize this with Ling today.
the test checks "docker daemon's" cpu usage when only system add-on pods are running, and the usage increases on all nodes. fluentd is one of the few pods that run on every node and the failure coincided with the fluentd version upgrade. On top of that, fluentd frequently fail its own liveness checks. Here are a few things that are worth checking IMO: Revert the PR (or run the test in a separate test cluster) to confirm whether fluentd is the culprit. Confirm whether the increase of "docker daemon" cpu usage is intended. Why is fluentd failing the liveness checks so frequently? Is that expected.

I've seen a few consecutive failures (although they were under the threshold, which is 3). Is the newer version of fluentd logging more? Docker daemon is responsible for writing the container stdout/stderr logs to the disk, and the increase of logs could cause cpu spikes. Just check the fluentd logs and see if there is anything abnormal.
/assign (since I can't assign it to )
After doing some investigation, we discovered that this was due to an update in Fluentd v0.14 in how it manages its buffers. I've submitted a PR that resolves the liveness probe which should prevent the extreme spikes in CPU when the fluentd process is forcefully shut down.
update? cc AI needed here
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
The two PRs that should resolve this issue have been approved and will be merged soon.


Question: inList $etcdName $myIP configEtcd $etcdName $myIP $cluster # set MINION IPs in kube-controller-manager <del> sed -i "s/MINION_IPS/${minionIPs}/g" default_scripts/kube-controller-manager </del> <ins> sed -i "s/MINION_IPS/${minionIPs}/g" work/default_scripts/kube-controller-manager </ins> cpMaster break ;;

I used the ubuntu-cluster setup scripts to build a mini testing environment. The first time I inputted a wrong config and when I reconfigured the environment by running the scripts again an error occured. I looked into the code and found the scripts hadn't deal with old configs. And of kube-proxy configs are not replaced.
Looks like this was fixed by .


Question: if i := bytes.Index(line, []byte(separator)); i == 0 { // We have a potential document terminator i += sep <del> after := line[i:] if len(strings.TrimRightFunc(string(after), unicode.IsSpace)) == 0 { if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err </del> <ins> trimmed := strings.TrimSpace(string(line[i:])) // We only allow comments and spaces following the yaml doc separator, otherwise we'll return an error if len(trimmed) > 0 && string(trimmed[0]) != "#" { return nil, YAMLSyntaxError{ err: fmt.Errorf("invalid Yaml document separator: %s", trimmed), </ins> } } <ins> if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err } </ins> } if err == io.EOF { if buffer.Len() != 0 {

Attempting to create three namespaces (foo, bar, baz). Notice the first YAML document separator has the comment after it on the same line. Output: Exit code is 0 and only and were created. It should either create all three namespaces or else it should fail with an error telling me my YAML is invalid. See Above. Same thing happens if I put random characters after the separator. For example: No error message is printed, exit code is 0, and the next resource is just silently ignored Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/assign
/sig api-machinery likely something that does not tolerate? you can try reproducing that with a unit test.
Thanks for the hint! I actually had a look at it and the Yaml document's termination logic seems to happen here: We currently assume that three dashes only terminate the document if followed by spaces (or by new line immediately), otherwise we keep scanning the document consider it as a single doc (including three dashes and any characters that follow). Ref: Doing that will make the yaml unmarshalling ignore everything that is followed by these three dashes (as it does not support unmarshalling multiple docs) To prove it: Will output: According to the Yaml Spec it's OK to have any character after these three dashes, but it will be considered part of the new doc already (as three dashes indicate seperation of Yaml directives from document's content) see One thing to note here is we ignore the use of three dots ("...") as a separator of one document's from another's Yaml directives. So for example if we had this kind of Yaml we would consider to be part of the first doc, which is wrong assumption according to the spec. To keep it simple though, I will sign a PR (including tests) later that will handle the three dashes and the followed characters to be a termination indicator and possible taken under consideration for the following (second) doc.
Thanks for having a look.
Let us know when you send a PR, thank you!

/triage accepted
I would love to have your opinions on the following: There are 3 fix stages to this bug as I see it: 1) We should split to different Yaml documents when three dashes appears, whether they followed by comments (or any other content), and not only spaces, so we would handle the strange behavior described in the bug description. 2) We should consider three dashes as part of the next document (and not only as a terminator - so for example in we won't be dropping the existence of This one is breaking because when three dashes appear at the end of the document i.e we would consider this file as containing 2 docs (the second is empty). 3) We need to take 3 dots under consideration as described above. This would also be breaking change obviously (and in many more ways in contrast to the 2nd change) and in my opinion needs to be filed as an entirely different issue Each one of these changes get us closer to an alignment state with the Yaml spec. What do you suggest to include for this PR, stage1, or stage 2 as well?
I'm in the process of . To keep it simple, the approach I settled on, at least so far, is to allow (and ignore) white space or comments, but any other content will result in an error. As you pointed out there is a lot more to the YAML syntax. However, it can be quite a bit more work trying to support all the possible things that are syntactically correct but not something that would be used in kubernetes, like . I also chose not to address because my impression was that this is mostly used for "streamed" YAML where there is no EOF. I actually didn't even know that existed before digging into the spec. I'm not saying it would be bad to support that, just that I don't think anyone uses it for their manifests so the "ROI" for making that change would be low.
my vote would be to not introduce breaking changes, because this code path is used by literally everyone with manifest files. note that the same code is used for JSON multi-doc splitting too. i think ideally we should not be splitting at any occurrence of . is the specification explicit about the being at position 0 of a new line?

if so the accurate way would be: split at add back a to item add everything after to item (excluding the delimiter ) but that might be breaking already. i think the current behavior is to drop empty docs and the start and end. applies to: resulting in only one doc. this is quite common in manifests. probably we shouldn't manage 3 dots because i haven't seen requests for it. also depends on what the spec says and if the split should happen at and not any . special casing only white space and comments after a makes sense for the time being, IMO.
I agree, breaking changes are dangerous here and should be done gradually (although you right we already apply empty docs as in this case: so taking things like as described above would not be actually breaking. Anyways, I think we'll act cautiously as a result of this change's sensitivity, and go with approach to this here by special casing whitespaces and comments. Thanks for the involvement.
PR is ready. ^


Question: /* Release : v1.9 Testname: Scheduler, resource limits <del> Description: Scheduling Pods MUST fail if the resource limits exceed Machine capacity. </del> <ins> Description: Scheduling Pods MUST fail if the resource requests exceed Machine capacity. </ins> */ framework.ConformanceIt("validates resource limits of pods that are allowed to run ", func() { WaitForStableCluster(cs, masterNodes)

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: ) func setDesiredConfiguration(initialConfig *kubeletconfig.KubeletConfiguration) { <del> initialConfig.EnforceNodeAllocatable = []string{"pods", "kube-reserved", "system-reserved"} </del> <ins> initialConfig.EnforceNodeAllocatable = []string{"pods", kubeReservedCgroup, systemReservedCgroup} </ins> initialConfig.SystemReserved = map[string]string{ string(v1.ResourceCPU):    "100m", string(v1.ResourceMemory): "100Mi",

/kind failing-test /sig node , , Failure: is the likely culprit. /assign cc
/kind e2e-test-failure
I'll take a look... Thanks!


Question: echo "Setting ${name}'s aliases to '${aliases}' (added ${ip})" # Attach ${ip} to ${name} gcloud compute instances network-interfaces update "${name}" --project "${PROJECT}" --zone "${zone}" --aliases="${aliases}" <del> run-gcloud-command "${name}" "${zone}" 'sudo ip route add to local '${ip}'/32 dev $(ip route | grep default | awk '''{print $5}''')' || true </del> <ins> gcloud compute instances add-metadata "${name}" --zone "${zone}" --metadata=kube-master-internal-ip="${ip}" run-gcloud-command "${name}" "${zone}" 'sudo /bin/bash /home/kubernetes/bin/kube-master-internal-route.sh' || true </ins> return $? }

If you create a private cluster via kube-up using provider GCE, you won't be able to restart your master nodes. If you restart you master, the nodes won't be able to connect to your master. This is due to this line - It gets executed on master on cluster-up, but it won't be re-executed on master restart. We should fix that, especially if we're to implement /sig scalability /priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /good-first-issue I'll ask whether someone in SIG Scalability public meeting would like to take on this one
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello, Can I start work on this one as my first issue? :)
I don't see anyone else claiming it, so your contributions are welcome :)
let me know if you need any explanations here
Thank you :) I've made some changes and assign you can you tell me is this will be good approach?
Thanks, I replied in the PR


Question: <ins> # Copyright 2016 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. TAG = e2e PREFIX = gcr.io/google_containers IMAGE = iperf all: push container: image image: docker build -t $(PREFIX)/${IMAGE} . # Build new image and automatically tag it as latest docker tag $(PREFIX)/${IMAGE} $(PREFIX)/${IMAGE}:$(TAG) # Add the version tag to the latest image push: image gcloud docker push $(PREFIX)/${IMAGE} # Push image tagged as latest to repository gcloud docker push $(PREFIX)/${IMAGE}:$(TAG) # Push version tagged image to repository (since this image is already pushed it will simply create or update version tag) clean: </ins>

This is a simple issue, no code required :) , just need to push the images/ iperf container to gcr . details merged a Dockerfile for building iperf containers. can be completed once we push to gcr. maybe you can help push the container since you reviewed the original pr ? That will allow me to do : which will finalize this iteration of the iperf network tests.
the image now exists, but produced a ton of complaints about automake-1.9 not existing on my system. Can you let me know if this was fatal (still seemed to build stuff)?
ok ill test onto my own docker repo and see
Thanks! You'll probably want to do
Ok. So, the container is working for me without building it.... so the GCR stuff is working
Super! That is what counts :)
ok ... and if we add to the Dockerfile, then it will output one-less error message, so that might be a good PR to put in. We can also upgrade to iperf3 after the iperf2 e2e runs a little while.


Question: rw := &kubeConfigReadWriter{ kubeConfigFilePath: tt.kubeConfigFilePath, } <del> if got := rw.Exists(); got != tt.want { </del> <ins> if got, _ := rw.Exists(); got != tt.want { </ins> t.Errorf("kubeConfigReadWriter.Exists() = %v, want %v", got, tt.want) } })

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: "//vendor:github.com/blang/semver", "//vendor:github.com/docker/engine-api/types", "//vendor:github.com/docker/engine-api/types/container", <ins> "//vendor:github.com/docker/go-connections/nat", </ins> "//vendor:github.com/golang/mock/gomock", "//vendor:github.com/stretchr/testify/assert", "//vendor:github.com/stretchr/testify/require",

This is in 1.6.0-beta.4: Some context: The line number suggests it is here: I think this is because dockershim now uses a protobuf object and Protocol is now an int. And is allowed in go (apparently) - is "A". And the comedic error is because Protocol implements If this is the error, there's something fishy going on because all our UDP tests should fail?
Aha ... I think it is only UDP hostports that would be "broken" by the issue. I wouldn't be surprised if we don't rely on any, and that we might not have any test coverage. I don't think this is a regression - I think this code has been there since the start of dockershim, but maybe CRI is now the default? I don't know how widely UDP hostports are used. Given how close we are to the planned release date, cc and
There is something wrong in dockershim's protocol parsing. Will file an PR to fix this. /assgin
/assign
Opened to fix this.
thanks for filing the issue. Thanks for quickly addressing the bug. We need node e2e tests to capture the kind of regression at the first place.


Question: fi if [ "$cert_ip" == "_use_azure_dns_name_" ]; then <del> cert_ip=$(hostname -f | awk -F. '{ print $2 }').cloudapp.net </del> <ins> cert_ip=$(uname -n | awk -F. '{ print $2 }').cloudapp.net </ins> use_cn=true fi

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: rc := common.NewDynamicResourceConsumer(rcName, common.KindDeployment, pods, allPodsCPU, memoryUsed, 0, perPodCPU, memoryLimit, f) defer rc.CleanUp() <del> rc.WaitForReplicas(pods) </del> <ins> rc.WaitForReplicas(pods, 15*time.Minute) </ins> metricsMap := map[string]bool{} pollingFunction := checkForMetrics(projectId, gcmService, time.Now(), metricsMap, allPodsCPU, perPodCPU)

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: for _, l := range strings.Split(string(f.Lines), "n") { if strings.Contains(l, fmt.Sprintf("-A %v", chainName)) { newRule := Rule(map[string]string{}) <del> for _, arg := range []string{Destination, Source, DPort, Protocol, Jump, ToDest, Recent, MatchSet} { </del> <ins> for _, arg := range []string{Destination, Source, DPort, Protocol, Jump, ToDest, Recent, MatchSet, SrcType} { </ins> tok := getToken(l, arg) if tok != "" { newRule[arg] = tok

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: This is discussed on When externalTrafficPolicy: Local is used, in GKE, external traffic coming in to the Load Balancer works correctly. But traffic that originates within the GKE cluster is broken, a TCP SYN is not responded to. My use case: I run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes. But I also access the docker-registry outside kubernetes. So my hostname for it is ''. In DNS this resolves to the LB IP. Actually, all 3 services have DNS names that resolve to the same LB IP, which then goes to the nginx ingress. so , , are generally addressed as LB-IP w/ HOSTNAME+SNI. Its 'astonishing' to not be able to access this DNS name when I'm 'inside' but can when I'm 'outside'. If I enable the IP transparency (because in my wiki I want the IP of the user who made the edit), then my gitlab-runner fails(since it cannot pull from the registry). Tim Hockin has this to say: and then: What you expected to happen: My expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public IP it should work. How to reproduce it (as minimally and precisely as possible): Configure a Load Balancer, an ingress, and 2 services. Have service 2 talk to service one using a DNS name that maps to the Load Balancer. Anything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-05-10T17:23:18Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): BUILDID=.89.0 NAME="Container-Optimized OS" KERNELCOMMITID= GOOGLECRASHID=Lakitu VERSIONID=66 PRETTYNAME="Container-Optimized OS from Google" VERSION=66 GOOGLEMETRICSPRODUCTID=26 HOMEURL="" ID=cos Kernel (e.g. ): Linux gke-k8s-default-pool--d3ft 4.14.22+ SMP Thu May 10 17:54:42 PDT 2018 x8664 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux Install tools: Others:
/sig network
Agree that this is weird and feels left-out. The fix is not complicated but I need an owner to carry it through testing.
if nobody is working on it can I take it as my first issue?.
Is anybody working on this issue. Seems like ipvs might have the same problem.
/assign
I believe I'm running into this problem and I think adding as Tim mentioned would fix it. are you working on this? If not I don't mind taking a stab at it :)
Thanks, Sorry I kinda forgot this thing. I was fixed this thing just like you said But encounter some problem when testing in gce environment, which caused by route rules. Sorry I forgot the detail, can you do some test in your environment?
It seems the pull request is stalled on this. Does anyone have any input?
Being relied on by istio
Following this.
Did this fix get lost in the shuffle?

Any chance getting this merged in? This also impacts Digital Ocean's Kubernetes + LB setups out of the box.
As far as I know it hasn't
Merge On Mon, Feb 18, 2019, 7:58 PM Jarred Capellman wrote:
still affecting me on gke. i want to enable the so that I can use the spam-blocking in wordpress (which is by ext ip), but I also want to refer to other services which may be internal to the cluster via their proper name (to get the `tls etc).
Same here.
I have more or less spelled out the fix, but I don't have teh bandwidth to polish it off right now. I am looking for volunteers...
I can take this given I've run into this a while back /assign
I was only able to reproduce this when trying to reach the public IP from the host network. Reaching the public IP from inside the pod network is working (as per this ). It's possible I missed a step in trying to reproduce this but wanted to confirm with you first. Are you seeing this for traffic originating only from host network or does it also apply for pods in the cluster network?
Hi. My situation. I have a DNS name (git.MYDOMAIN). It runs on a self-hosted gitlab via nginx-ingress via LoadBalancer in GKE. Other pods of mine sometimes reference https://git.MYDOMAIN. They may happen to run in the same cluster as git.MYDOMAIN, they may run in a different cluster. When they run in the same cluster on GKE, they fail if is enabled. Specifically, since the DNS resolves to (My external Loadbalancer IP) an IP outside of GKE, it goes out, hits the LB, and then somewhere along the reverse path is lost. I have calico enabled (as per GKE managed). I am not using Istio. I am using a cluster that predates the new private IP ones. I have not tried disabling the LoadBalancer and just using node ip for the ingress. I have not tried to see if there is any difference if the Pod trying to reach my git is on the same node as it or not.

Based on the proxy rules (may be different at this point cause I'm looking at master), I don't think your traffic is even reaching the LB, it's just being dropped by the iptable rules on the host. My original question was, are the pods trying to reach the same-cluster external IP have , if yes, can you reproduce on pods not on host network?
I do not have set.
Are you still on v1.10?
this is the most recent version on gke afaik.
I opened a fix here based on the comments and some testing I did on my GKE cluster. It needs more validation though
Again, I wasn't able to reproduce this for pods that are within the cluster CIDR (but I could on the host network). The rule that allows for pods inside the cluster CIDR to still reach pods behind an LB is this: Are you able to check if a rule like ^ exists on your cluster?
Here's an example, right now (www is hosted on ingress w/ loadbalancer w/ local policy). Note: this example is from the node (not a pod).
Here's the full rules:
^ can you show me the same test from within a pod (not host network)? I have a fix for within a node but I'd like to make sure it's reproducible inside a pod before I dig deeper
Interestingly its working in a Pod on this cluster now. (?!) I guess some previous upgrade of k8s on gke has altered it? on cannot reach on on can reach. hmmm.
ok w/o the node being able to reach this i'm still blocked (since my container registry might run there, and thus kubelet cannot reach, so my gitlab CI pipeline cannot reach). but, i guess, now the pods w/o hostnetwork can, so some progress
Thanks for checking should fix the internal reachability issue from the host. Though it requires SNAT.
Hello Maybe better way is to check if and then skip to KUBE-SERVICES chain? So, request will be sent to loadbalancer and source IP will be preserved. Can you try this?
Solution described above will affect traffic from pods, it will not be . Then maybe we can replace with something to send traffic to loadbalancer?

I think this will break existing implementations, the fwchain is what routes the LB IPs to the correct service chain. Not all LB implementations (think MetalLB) depend on node port for routing
I tried to exclude node IP with from on node without local endpoints and it works! What do you think about this solution? I tested with:
I believe there's a similar rule for that already but it doesn't check source IP If you jump to FW chain, you end up in XLB chain which drops traffic if there's no local endpoints. Does your test also work on nodes where there's no local endpoints? Can you also verify that your traffic actually goes to the LB and is not being masqueraded through the service chain?
Yes, I just source IP check to this rule on node without local endpoints, so local traffic doesn't jump to FW, it doesn't drops in XLB and it goes to LB. Of course, if you do this on node with local endpoints you will not reach that endpoint (at least when using Metallb). This solution doesn't work if nodes haven't network access to LB. Your solution haven't this problem. For me it is more important then saving source IP, so now I think your solution is better. But why you add jump to MARK-MASQ before jump to SVC when each SEP chain already contains jump to MARK-MASQ? I tested your solution with:
Good point, I think might be redundant, will update the PR soon :)


Question: if err != nil { return nil, fmt.Errorf("Failed to create tls config: %v", err) } <del> if tlsConfig != nil { </del> <ins> if url.Scheme == "https" { </ins> url.Scheme = "wss" <del> if !strings.Contains(url.Host, ":") { url.Host += ":443" } </del> } else { url.Scheme = "ws" <del> if !strings.Contains(url.Host, ":") { url.Host += ":80" } </del> } headers, err := headersForConfig(config, url) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When running e2e conformance tests against a cluster using a public HTTPS endpoint for the apiserver, you get the following errors: What you expected to happen: The API server endpoint is so the e2e tests should use and connect on port instead of . How to reproduce it (as minimally and precisely as possible): Run e2e conformance tests against a cluster with the apiserver exposed on a public endpoint with a public CA. Anything else we need to know?: When looking at the code here: It looks like the tlsConfig is not correctly derived from the URL. Diving into I see that this uses deep down: And this seem to return unless the kubeconfig has a CA or other explicit TLS config configured: I believe if this check also checked the of the URL, then it would be correct: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing