Question: }, )).To(Succeed(), "wait for pod %q to success", name) } <ins> // MatchContainerOutput gest output of a container and match expected regexp in the output. func (c *PodClient) MatchContainerOutput(name string, containerName string, expectedRegexp string) error { f := c.f output, err := GetPodLogs(f.ClientSet, f.Namespace.Name, name, containerName) if err != nil { return fmt.Errorf("failed to get output for container %q of pod %q", containerName, name) } regex, err := regexp.Compile(expectedRegexp) if err != nil { return fmt.Errorf("failed to compile regexp %q: %v", expectedRegexp, err) } if !regex.MatchString(output) { return fmt.Errorf("failed to match regexp %q in output %q", expectedRegexp, output) } return nil } </ins>

https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
the for this issue looks pretty good. We could probably close this issue.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Looks like more recent flakes:
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Adding labels for team node as is failure is part of the node e2e tests. All issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. Given the test flakes very infrequently and doesn't seem to be associated with a particular code change Could either or someone from please advise as to whether you believe this issue is actually a release blocker and I'll update the issue. Any updates from you would also be greatly appreciated.Thanks! cc:
The test was introduced by which causes some flakiness: Re-assign this to Lantao, I think the issue should be in the test itself, and non-resource-blocker. But I temporarily marked it as release blocker after your triage. Thanks!

This is a issue in the test itself. I've sent a fix for this
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: GO_OUT="${KUBE_ROOT}/_output/local/bin/${host_os}/${host_arch}" APISERVER_LOG=/tmp/kube-apiserver.log <del> "${GO_OUT}/kube-apiserver"  </del> <ins> sudo "${GO_OUT}/kube-apiserver"  </ins> -v=${LOG_LEVEL}  --address="${API_HOST}"  --port="${API_PORT}"

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


Question: // equivalent returns true if the two services are equivalent.  Fields which are expected to differ between // federated services and the underlying cluster services (e.g. ClusterIP, NodePort) are ignored. <del> func equivalent(federationService, clusterService v1.Service) bool { clusterService.Spec.ClusterIP = federationService.Spec.ClusterIP for i := range clusterService.Spec.Ports { clusterService.Spec.Ports[i].NodePort = federationService.Spec.Ports[i].NodePort </del> <ins> func equivalent(clusterService, federationService v1.Service) bool { federationService.Spec.ClusterIP = clusterService.Spec.ClusterIP for i := range federationService.Spec.Ports { federationService.Spec.Ports[i].NodePort = clusterService.Spec.Ports[i].NodePort } if federationService.Spec.ExternalTrafficPolicy == v1.ServiceExternalTrafficPolicyType("") { federationService.Spec.ExternalTrafficPolicy = clusterService.Spec.ExternalTrafficPolicy </ins> } if federationService.Name != clusterService.Name || federationService.Namespace != clusterService.Namespace {

Test pull-kubernetes-federation-e2e-gce for PR is failing. pull-kubernetes-federation-e2e-gce https://k8s- pull-kubernetes-federation-e2e-gce test this
/sig federation /assign
cc
Details on the types of failures and frequency: In the last two days in particular, the following tests have been timing out: [] Federated Services [Feature:Federation] with clusters Federated Service should create and update matching services in underlying clusters [] Federated Services [Feature:Federation] with clusters DNS should be able to discover a federated service [] Federated Services [Feature:Federation] with clusters Federated Service should recreate service shard in underlying clusters when service shard is deleted cc:
Thanks. And yes we are aware of it. is working on a fix - PR .
Hi I'm trying to find out how to follow up on this. Will there be some sort of notification for the PRs affected this?
I just re-queue that test on your PR. note that the test isn't blocking, so your PR can be reviewed/merged without this test passing.
good to know. Thanks!
you can't re-open an issue unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/reopen
Reopening to have an issue to track this. This was thought to be resolved, and was made blocking in There's still something up, and it was reverted in I believe sig-federation is going to look at this more next week in order to figure out a more permanent solution to this.
Closing this in favor of issue which has linked runs which more accurately describe the problem.


Question: HardPodAffinitySymmetricWeight: 1, HealthzBindAddress:             "", // defaults empty when not running from config file MetricsBindAddress:             "", // defaults empty when not running from config file <ins> DebuggingConfiguration: componentbaseconfig.DebuggingConfiguration{ EnableProfiling:           true, EnableContentionProfiling: true, }, </ins> LeaderElection: kubeschedulerconfig.KubeSchedulerLeaderElectionConfiguration{ LeaderElectionConfiguration: componentbaseconfig.LeaderElectionConfiguration{ LeaderElect:       true,

What would you like to be : Enable profiling by default in the scheduler. We can do this by defaulting the CC flags to true. Why is this needed: It is useful when debugging the performance of a running cluster. API server already enables profiling by default. /sig scheduling /help /milestone v1.17
/assign /remove-help


Question: }) } } <ins> func TestUpdatePodInCache(t *testing.T) { ttl := 10 * time.Second nodeName := "node" tests := []struct { name   string oldObj interface{} newObj interface{} }{ { name:   "pod updated with the same UID", oldObj: withPodName(podWithPort("oldUID", nodeName, 80), "pod"), newObj: withPodName(podWithPort("oldUID", nodeName, 8080), "pod"), }, { name:   "pod updated with different UIDs", oldObj: withPodName(podWithPort("oldUID", nodeName, 80), "pod"), newObj: withPodName(podWithPort("newUID", nodeName, 8080), "pod"), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { stopCh := make(chan struct{}) defer close(stopCh) schedulerCache := cache.New(ttl, stopCh) schedulerQueue := queue.NewPriorityQueue(nil) sched := &Scheduler{ SchedulerCache:  schedulerCache, SchedulingQueue: schedulerQueue, } sched.addPodToCache(tt.oldObj) sched.updatePodInCache(tt.oldObj, tt.newObj) pod, err := sched.SchedulerCache.GetPod(tt.newObj.(*v1.Pod)) if err != nil { t.Errorf("Failed to get pod from scheduler: %v", err) } if pod.UID != tt.newObj.(*v1.Pod).UID { t.Errorf("Want pod UID %v, got %v", tt.newObj.(*v1.Pod).UID, pod.UID) } }) } } func withPodName(pod *v1.Pod, name string) *v1.Pod { pod.Name = name return pod } </ins>

What happened: pod can't be removed from scheduler cache when be deleted, logs in kube-scheduler is below: I1128 16:41:50. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:50. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:53. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:53. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:57. ] successfully acquired lease kube-system/kube-scheduler I1128 16:41:57. ] caches populated E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it What you expected to happen: pod in scheduler cache can be deleted successfully when pod is deleted How to reproduce it (as minimally and precisely as possible): main steps to reproduce this bug is these: scheduler run in leader-elect=true modeschedulerA is the leader scheduler, and schedulerB is an assistant.

pod1 with "namespace1/name1" and it runs on Node1 down the network between schedulerB and kube-apiserver pod1 and create pod2 with same name, ie "namespace1/name1" and pod2 been assigned to Node1 by schedulerA again. off schedulerA and recover the network between schedulerB and kube-apiserver, and then schedulerB become the leader scheduler. pod2. this time log in schedulerB will print this: E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it this time, if I create a new pod3 with podAntiAffinity to pod1. the result is pod3 can't been assigned to Node1 because pod1 is still in schedulerB's scheduler cache. Anything else we need to know?: the reason of this bug is below: since PR(), podKey in scheduler cache use pod's UID instead of namespace/name however, the key in SharedIndexInformer still use namespace/name in this upper case, delete the pod and create a same name pod may be delivered as one update notification.( see disscusion ) scheduler received pod update notification and then print below log. E1128 16:42:11. ]

scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/sig scheduling
Since has been marked resolved (fix available in 1.17), does that mean this issue is expected to go away in 1.17? Could you please give a try?
, fix for just resolved the case add event may be lost after deletion event. It doesn't solve the problem that a delete followed by an add may be delivered as one update notification and it's thought normal from MikeSpreitzer's comment. The main cause of this problem is scheduler cache use Pod UID as key while sharedInformer use namespace/name. Still, I also test with the latest kubernetes code. The bug of this issue reappeared.
Thanks I a comment in the PR, and a test is necessary to reproduce the issue and verify the fix.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale ping there is an unresolved comment in the PR, do you have a chance to take a look?


Question: This rule maintains the invariant that all JSON/YAML keys are fields in API objects. The only exceptions are pure maps in the API (currently, labels, selectors, and annotations), as opposed to sets of subobjects. <ins> #### Constants Some fields will have a list of allowed values (enumerations). These values will be strings, and they will be in CamelCase, with an initial uppercase letter. Examples: "ClusterFirst", "Pending", "ClientIP". </ins> ### Lists and Simple kinds

The api conventions document should provide guidance on values for constants and naming of -like fields like .
Yes, apparently we didn't document the CamelCase rule.


Question: } } } <ins> func TestCheckDiskExists(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() ctx, cancel := getContextWithCancel() defer cancel() testCloud := GetTestCloud(ctrl) common := &controllerCommon{ location:              testCloud.Location, storageEndpointSuffix: testCloud.Environment.StorageEndpointSuffix, resourceGroup:         testCloud.ResourceGroup, subscriptionID:        testCloud.SubscriptionID, cloud:                 testCloud, vmLockMap:             newLockMap(), } // create a new disk before running test newDiskName := "newdisk" newDiskURI := fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/disks/%s", testCloud.SubscriptionID, testCloud.ResourceGroup, newDiskName) fDC := newFakeDisksClient() rerr := fDC.CreateOrUpdate(ctx, testCloud.ResourceGroup, newDiskName, compute.Disk{}) assert.Equal(t, rerr == nil, true, "return error: %v", rerr) testCloud.DisksClient = fDC testCases := []struct { diskURI        string expectedResult bool expectedErr    bool }{ { diskURI:        "incorrect disk URI format", expectedResult: false, expectedErr:    true, }, { diskURI:        "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Compute/disks/non-existing-disk", expectedResult: false, expectedErr:    false, }, { diskURI:        newDiskURI, expectedResult: true, expectedErr:    false, }, } for i, test := range testCases { exist, err := common.checkDiskExists(ctx, test.diskURI) assert.Equal(t, test.expectedResult, exist, "TestCase[%d]", i, exist) assert.Equal(t, test.expectedErr, err != nil, "TestCase[%d], return error: %v", i, err) } } func TestFilterNonExistingDisks(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() ctx, cancel := getContextWithCancel() defer cancel() testCloud := GetTestCloud(ctrl) common := &controllerCommon{ location:              testCloud.Location, storageEndpointSuffix: testCloud.Environment.StorageEndpointSuffix, resourceGroup:         testCloud.ResourceGroup, subscriptionID:        testCloud.SubscriptionID, cloud:                 testCloud, vmLockMap:             newLockMap(), } // create a new disk before running test diskURIPrefix := fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/disks/", testCloud.SubscriptionID, testCloud.ResourceGroup) newDiskName := "newdisk" newDiskURI := diskURIPrefix + newDiskName fDC := newFakeDisksClient() rerr := fDC.CreateOrUpdate(ctx, testCloud.ResourceGroup, newDiskName, compute.Disk{}) assert.Equal(t, rerr == nil, true, "return error: %v", rerr) testCloud.DisksClient = fDC disks := []compute.DataDisk{ { Name: &newDiskName, ManagedDisk: &compute.ManagedDiskParameters{ ID: &newDiskURI, }, }, { Name: pointer.StringPtr("DiskName2"), ManagedDisk: &compute.ManagedDiskParameters{ ID: pointer.StringPtr(diskURIPrefix + "DiskName2"), }, }, { Name: pointer.StringPtr("DiskName3"), ManagedDisk: &compute.ManagedDiskParameters{ ID: pointer.StringPtr(diskURIPrefix + "DiskName3"), }, }, { Name: pointer.StringPtr("DiskName4"), ManagedDisk: &compute.ManagedDiskParameters{ ID: pointer.StringPtr(diskURIPrefix + "DiskName4"), }, }, } filteredDisks := common.filterNonExistingDisks(ctx, disks) assert.Equal(t, 1, len(filteredDisks)) assert.Equal(t, newDiskName, *filteredDisks[0].Name) disks = []compute.DataDisk{} filteredDisks = filterDetachingDisks(disks) assert.Equal(t, 0, len(filteredDisks)) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: check disk status before disk azure disk There is little possibility that attach disk would cost 10min, and disk could be attached, deleted in same time, that would cause race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: // OpenLocalPort fakes out the listen() and bind() used by syncProxyRules // to lock a local port. <del> func (f *fakePortOpener) OpenLocalPort(lp *utilproxy.LocalPort) (utilproxy.Closeable, error) { </del> <ins> func (f *fakePortOpener) OpenLocalPort(lp *utilproxy.LocalPort, isIPv6 bool) (utilproxy.Closeable, error) { </ins> f.openPorts = append(f.openPorts, lp) return nil, nil }

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: progressReporter.ProcessSpecReport(report) }) <ins> var _ = ginkgo.ReportBeforeSuite(func(report ginkgo.Report) { progressReporter.SetTestsTotal(report.PreRunStats.SpecsThatWillRun) }) </ins> var _ = ginkgo.ReportAfterSuite("Kubernetes e2e suite report", func(report ginkgo.Report) { var err error // The DetailsRepoerter will output details about every test (name, files, lines, etc) which helps

Our team uses sonobuoy to run the conformormance tests on a cluster and displays the results to the users. We have been using conformance image 1.21.x until now and it sends back progress data to sonobuoy pod and our service reads it from the sonobuoy pod. The progress data included the total number of tests to be run and the completed tests. For 1.21 conformance we were receiving this data properly but when switching to 1.25.x conformance we are getting the total tests as zero. This is causing issues in our UI. Not sure what version of conformance image introduced this but we were using 1.21 and started seeing this issue after moving to 1.25.x. The conformance image versions in between 1.21 -1.25 are not CVE free so we cannot use them. Total number of tests should be non-zero for conformance image 1.25.x. Ideally i think it should come back as 360. provision a 1.25 kubernetes kind (or anything else) cluster Install sonobuoy Run conformance scan on the cluster Check the annotation on the sonobuoy pod . This is set by sonobuoy based on the data it gets back from the e2e conformance pod. The total tests comes as zero I just found it on 1.25.x conformance. Not really sure when this was introduced. Need to make sure back dated versiosn also have this fix in case they have this issue. <details25 cluster with 1.25.x conformance with any sonobuoy version </details<detailsANY </details<details</details<details</details<details</details<detailsConformacne e2e plugin </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
FYI
Yeah, it seems that it gets the info here I vaguely remember something about a progress custom reporter that was by sonobuoy and we didn't know what was used for
You remember correctly: It looks like that TODO didn't get addressed? I thought it had been. the other thing that was unclear regarding sonobuoy was whether it reads these test statistics from stderr or gets them via HTTP. We removed the stderr output (I think in 1.26) but kept (). This seems to be backed by (sonobuoy sets the parameter), but it would be good to get definite confirmation. Also, does Sonobuoy run tests in parallel? If it does, then these statistics are probably useless or at least will need considerable post-processing.
We depend on for this.
/assign I will take a look as has been fixed and closed
/reopen The fix went in, but let's keep the issue open for a while longer until we really know that it now works as expected. /assign
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
/remove-lifecycle rotten I discussed this further with (see also ) and while a final decision is pending, I am inclined to remove the progress reporting feature because I suspect that it is conceptually broken (to be confirmed) and lacks in-tree support (unit tests, owner). Before I do that, can you clarify: Are you running tests in parallel and does progress reporting work properly, i.e. does it go from 0 to 100%? When you said, "causing issues in our UI", what UI is that?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Ping Please see
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Update internal cached podDevices state. m.mutex.Lock() for dev := range allocDevices { <ins> if m.allDevices[resource][dev].Topology == nil || len(m.allDevices[resource][dev].Topology.Nodes) == 0 { allocDevicesWithNUMA[0] = append(allocDevicesWithNUMA[0], dev) continue } </ins> for idx := range m.allDevices[resource][dev].Topology.Nodes { node := m.allDevices[resource][dev].Topology.Nodes[idx] allocDevicesWithNUMA[node.ID] = append(allocDevicesWithNUMA[node.ID], dev)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: <ins> apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: default annotations: storageclass.beta.kubernetes.io/is-default-class: "true" labels: kubernetes.io/cluster-service: "true" provisioner: kubernetes.io/azure-disk </ins>

We'd like to deprecate alpha provisioning in 1.6. In order to make current deployments working after upgrade to 1.6 we need to deploy a default StorageClass as part of installation/upgrade on Azure so it will automatically provision volumes for PVCs marked with annotation - the annotation will be ignored and default storage class must provision the volume instead. On GCE and AWS, we use addon manager to install the default storage class, see Note that some extra code is probably needed to actually put the storage class into addons directory during installation, e.g. for manual script-based installation on GCE or for salt-based installation. Please make sure that: A default StorageClass is deployed during "default" installation. It would be nice if admin could turn this off during installation (e.g. as on GCE) If any additional manual steps are needed to deploy the default storage class make sure they're documented in appropriate installation guide.
Assigning to random Microsoft personnel for faster routing to proper person.
I believe this is fixed, correct? If so, I will close it...
Yes, it's fixed.


Question: # do_start() { <ins> # Avoid a potential race at boot time when both monit and init.d start # the same service PIDS=$(pidof $DAEMON) for PID in ${PIDS}; do kill -9 $PID done </ins> # Raise the file descriptor limit - we expect to open a lot of sockets! ulimit -n 65536

During an e2e test run, on GCE, I ssh-ed into the box. After the reboot test runs, it appears that something is trying to start kubelet every 2 minutes after the reboot: It does appear that it is monit restarting the process: I do only have one copy of kubelet running, because the second kubelet fails to start:
The direct cause of monit thinking the kubelet is down is that does not match. Also, keeps changing (every two minutes?) My hypothesis is that the second kubelet writes the pid, because it gets pretty far in its boot-up. And thereafter, monit sees it as down so keeps trying to start more. And so the root cause is that first second kubelet start, which I think is down to this: My kubelet is currently listening to that url though: My guess is that kubelet is too slow to start listening to /healthz
Oh! ... or it's a simple race with init.d starting the services; monit vs kube-proxy. If monit starts kube-proxy before init.d starts kube-proxy =double start.
/cc
Ok, there is a race when monit is used to monitor services that are also started at boot time via init.d, in our ecosystem, such as kubelet, kube-proxy, and docker. Especially when a service is slow to start, monit can assume that the service is not running and possibly try to start it and raise an issue while the service is already about to start or already in its startup sequence. In this case, there is a stale pidfile which is used by monit during the service validation periodically (currently configured every 2 mins), thus run into the issue reported by here. A clean way to fix this is starting those kubernetes related services requiring monitoring by monit directly by monit. The issue is that monit is not installed by default in our gce images for now. My proposed solution is that for each dostart call in /etc/init.d/${servicedaemon}, checking if there is existing process running. If it does, kill it before calling start-stop-daemon --start Will send pr shortly.


Question: t.Errorf("unexpected error: %v", err) } if len(times) != 2 { <del> t.Errorf("expected 2 start times, got: , got: %v", times) </del> <ins> t.Errorf("expected 2 start times, got: %v", times) </ins> } else { if !times[0].Equal(T1) { t.Errorf("expected: %v, got: %v", T1, times[0])

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen


Question: } } } <ins> func TestExtractDeviceAndPrefix(t *testing.T) { devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00" lun := "-lun-0" device, prefix, err := extractDeviceAndPrefix("/var/lib/kubelet/plugins/kubernetes.io/iscsi/" + devicePath + lun) if err != nil || device != (devicePath+lun) || prefix != devicePath { t.Errorf("extractDeviceAndPrefix: expected %s and %s, got %v %s and %s", devicePath+lun, devicePath, err, device, prefix) } } func TestExtractPortalAndIqn(t *testing.T) { devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0" portal, iqn, err := extractPortalAndIqn(devicePath) if err != nil || portal != "127.0.0.1:3260" || iqn != "iqn.2014-12.com.example:test.tgt00" { t.Errorf("extractPortalAndIqn: got %v %s %s", err, portal, iqn) } } </ins>

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> ) func TestRunExposeService(t *testing.T) {

This was broken by , and therefore broken in as well. Any fix should be offered as a cherry pick so "make release" is clean there.
I went ahead and worked something up for this this morning, to get Shippable builds back on the rails.


Question: Port:    uint16(portNum), } <del> klog.V(5).Infof("Using graceful delete to delete: %v", delDest) </del> <ins> klog.V(5).Infof("Using graceful delete to delete: %v", uniqueRS) </ins> err = proxier.gracefuldeleteManager.GracefulDeleteRS(appliedVirtualServer, delDest) if err != nil { klog.Errorf("Failed to delete destination: %v, error: %v", delDest, err)

What happened: On pods with a lot of DNS traffic, many DNS queries to the DNS ClusterIP fail during a rolling update of the DNS pods (in our case, we use coredns). What you expected to happen: Almost all DNS queries should succeed during a rolling update. How to reproduce it (as minimally and precisely as possible): Create a coredns (this would probably be the same with kube-dns) service backed by 2 pods Access this service by creating a lot of DNS queries, for example: Wait for the IPVS connection table to contain a lot of DNS connections (a few minutes) Delete a DNS pod A significant number of queries will timeout. After 5mn (depending in the IPVS timeout configuration) everything will work ok again. Anything else we need to know?: This comes from the current logic in the graceful termination code: The current logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed. In addition, will not always work here because IPVS only removes connections when a packet is processed (in the case of DNS once the client receives an answer, no more packets are sent). Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign
I think there is also an issue with the way we clear the conntrack for UDP We call in : If I understand correctly this will clear conntrack entries matching and before graceful termination is done. I think we should only remove the conntrack entries when we delete the real server. In addition, we only clear the conntrack entries for <clusterIP, I think we should also clean them for Node IPs (Nodeports) and External IPs. What do you think?
I think we still need to keep this issue open even though we have merged - I assume they are different issues.

yes I think the PR is a great improvement but we probably need a few other small fixes (like the conntrack thing I mentioned)
The fix from the PR is not perfect yet for DNS traffic when qps is high (but it's definitely an improvement for UDP traffic which didn't support any kind of graceful termination). For example: Will generate 5000qps and ~500 new entries in the IPVS table per second. With a UDP timeout of 30s this leads to entries maximum. If we have 2 pods, when we delete one we have to wait for the ~7500 entries to expire, given the timeout (30s) many ports will be reused (which resets the timeout) and when the coredns pod finishes its lameduck period, we lose queries. 5000 qps against 2 coredns pods is high and unlikely but to decrease the probability of DNS queries failing, we can either: decrease the UDP timeout (in this example if set to 10s, we dont lose a single packet) increase the lameduck period (I tested with UDP timeout 30s and lameduck 180s and it worked great) (of course with more pods the impact is less important: what matters is the number of ports assigned to Terminating pods) I think being able to control the UDP timeout is very important, the default (300s) is very high (see: ) (I forked dnstrace to add the parameter: , I'll try and get it merged in the main repository)
can you please help confirm our issue? in our system, our dns query is quite often, like 1 query per 10 seconds. in this case, we found that if we restart coredns, the old IPVS rules are never deleted on some of our nodes, which caused a lot of failed dns queries. is this because the graceful termination mechanism introduced here? In order to delete the RS and remove the ipvs rules, we have to wait the DNS query expire, but before the old dns query expires, new dns query came in, so grace termination will never see and old ipvs rules are never cleaned up, causing dns query issue.

if this is the case, I would suggest to introduce a force termination mechanism
: it depends how you query DNS: if you have a persistent resolution socket, the IPVS backend may never expire but in that case your resolver should timeout and reconnect which will use an active backend. If you use a standard resolver (glibc for instance) each query will get a new connection so you will reach an active coredns unless you query frequently enough so you end up reusing a same source port associated with a removed backend (but below 100 queries/second this is very unlikely). I have ideas to address both use cases: configure more aggressive timeouts (in particular for UDP) and make them configurable (see issue: ) forcefully remove real servers (coredns in your case) that are still tracked by IPVS after X minutes (today we only delete them when they have 0 active and 0 inactive connections which may never happen depending on your workload) may have ideas on this topic too
DNS resolution through kube-proxy is tricky for several reasons regardless of the proxy mode (for iptables for instance, many people have conntrack issues when doing a lot of queries). In our case, we are experimenting with node local coredns caches (you can also have a look at which is an implementation of this idea)
isn't 100 queries/second quite common for a site? QPS can be thousands nowadays. we did not do special configuration for our dns so it should be using glibc. I think what you mentioned about reusing the same source port should be our issue here. by checking the output of ipvsadmin, it seems the source port is reused for different udp queries <img width="517" alt="" src="https://user-"do you have any other suggestions on this issue?
btw, forcefully remove real servers is what I want here.
I meant 100 qps for a single client (which can definitely happen though) Yes it is very likely that you have the issue I mentioned Until we implement the solution I mentioned natively, you can try changing the ipvs udp timeout on your instances. The default are: You can change udp timeout to 30s (or even lower depending on your connection rate): (0 here means do not touch tcp/tcpfin)


Question: - linters: - ginkgolinter text: use a function call in (Eventually|Consistently) <ins> # SSA Extract calls are allowed in tests. - linters: - forbidigo text: should not be used because managedFields was removed path: _test.go$ </ins> linters: disable-all: true # not disabled in golangci-strict.yaml enable: # please keep this alphabetized and in sync with golangci-strict.yaml <ins> - forbidigo </ins> - ginkgolinter - gocritic - govet

Add a code static check for to report using server-side-apply extract With optimization of and , the will be removed in and , and maybe more of other components in the future, using Extract is problematic. It's safer to add this lint check.
/cc
/sig api-machinery
/sig scheduling /sig apps
/assign /triage accepted


Question: //   user: //     token: red-token } <ins> func TestDeduplicate(t *testing.T) { testCases := []struct { src    []string expect []string }{ { src:    []string{"a", "b", "c", "d", "e", "f"}, expect: []string{"a", "b", "c", "d", "e", "f"}, }, { src:    []string{"a", "b", "c", "b", "e", "f"}, expect: []string{"a", "b", "c", "e", "f"}, }, { src:    []string{"a", "a", "b", "b", "c", "b"}, expect: []string{"a", "b", "c"}, }, } for _, testCase := range testCases { get := deduplicate(testCase.src) if !reflect.DeepEqual(get, testCase.expect) { t.Errorf("expect: %v, get: %v", testCase.expect, get) } } } </ins>

/kind bug /sig cli What happened: The same path was multiple times to KUBECONFIG env variable. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-05T00:40:04Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"} Cloud provider or hardware configuration**: Irrelevant OS (e.g. from /etc/os-release): MacOS Sierra 10.12.6 Kernel (e.g. ): Darwin Kernel Version 16.7.0 Install tools: brew install kubernetes-cli Others:
I would help fix it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: <match reform.**> type record_reformer enable_ruby true <del> tag kubernetes.${tag_suffix[3].split('-')[0..-2].join('-')} </del> <ins> tag kubernetes.${tag_suffix[4].split('-')[0..-2].join('-')} </ins> </match> <source> type tail format none <del> path /varlog/salt/minion pos_file /varlog/gcp-salt.pos </del> <ins> path /var/log/salt/minion pos_file /var/log/gcp-salt.pos </ins> tag salt </source> <source> type tail format none <del> path /varlog/startupscript.log pos_file /varlog/gcp-startupscript.log.pos </del> <ins> path /var/log/startupscript.log pos_file /var/log/gcp-startupscript.log.pos </ins> tag startupscript </source> <source> type tail format none <del> path /varlog/docker.log pos_file /varlog/gcp-docker.log.pos </del> <ins> path /var/log/docker.log pos_file /var/log/gcp-docker.log.pos </ins> tag docker </source> <source> type tail format none <del> path /varlog/etcd.log pos_file /varlog/gcp-etcd.log.pos </del> <ins> path /var/log/etcd.log pos_file /var/log/gcp-etcd.log.pos </ins> tag etcd </source> <source> type tail format none <del> path /varlog/kubelet.log pos_file /varlog/gcp-kubelet.log.pos </del> <ins> path /var/log/kubelet.log pos_file /var/log/gcp-kubelet.log.pos </ins> tag kubelet </source> <source> type tail format none <del> path /varlog/kube-apiserver.log pos_file /varlog/gcp-kube-apiserver.log.pos </del> <ins> path /var/log/kube-apiserver.log pos_file /var/log/gcp-kube-apiserver.log.pos </ins> tag kube-apiserver </source> <source> type tail format none <del> path /varlog/kube-controller-manager.log pos_file /varlog/gcp-kube-controller-manager.log.pos </del> <ins> path /var/log/kube-controller-manager.log pos_file /var/log/gcp-kube-controller-manager.log.pos </ins> tag kube-controller-manager </source> <source> type tail format none <del> path /varlog/kube-scheduler.log pos_file /varlog/gcp-kube-scheduler.log.pos </del> <ins> path /var/log/kube-scheduler.log pos_file /var/log/gcp-kube-scheduler.log.pos </ins> tag kube-scheduler </source> <del> <match **> </del> <ins> # We use 2 output stanzas - one to handle the container logs and one to handle # the node daemon logs, the latter of which explicitly sends its logs to the # compute.googleapis.com service rather than container.googleapis.com to keep # them separate since most users don't care about the node logs. <match kubernetes.**> </ins> type google_cloud # Set the chunk limit conservatively to avoid exceeding the GCL limit # of 2MB per write request. buffer_chunk_limit 512K <del> # Cap buffer memory usage to 512KB/chunk * 128 chunks = 65 MB buffer_queue_limit 128 </del> <ins> # Cap the combined memory usage of this buffer and the one below to # 512KB/chunk * (96 + 32) chunks = 65 MB buffer_queue_limit 96 # Never wait more than 5 seconds before flushing logs in the non-error case. </ins> flush_interval 5s # Never wait longer than 5 minutes between retries. max_retry_wait 300 # Disable the limit on the number of retries (retry forever). disable_retry_limit </match> <ins> # Keep a smaller buffer here since these logs are less important than the user's # container logs. <match **> type google_cloud detect_subservice false buffer_chunk_limit 512K buffer_queue_limit 32 flush_interval 5s max_retry_wait 300 disable_retry_limit </match> </ins>

When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.


Question: return ps } <del> func (p *petSetTester) execInPets(ps *apps.PetSet, cmd string) { </del> <ins> func (p *petSetTester) checkMount(ps *apps.PetSet, mountPath string) error { for _, cmd := range []string{ // Print inode, size etc fmt.Sprintf("ls -idlh %v", mountPath), // Print subdirs fmt.Sprintf("find %v", mountPath), // Try writing fmt.Sprintf("touch %v", filepath.Join(mountPath, fmt.Sprintf("%v", time.Now().UnixNano()))), } { if err := p.execInPets(ps, cmd); err != nil { return fmt.Errorf("failed to execute %v, error: %v", cmd, err) } } return nil } func (p *petSetTester) execInPets(ps *apps.PetSet, cmd string) error { </ins> podList := p.getPodList(ps) for _, pet := range podList.Items { stdout, err := framework.RunHostCmd(pet.Namespace, pet.Name, cmd) <del> ExpectNoError(err) </del> framework.Logf("stdout of %v on %v: %v", cmd, pet.Name, stdout) <ins> if err != nil { return err } </ins> } <ins> return nil </ins> } func (p *petSetTester) saturate(ps *apps.PetSet) {

Eg: Suspect it's because of:
If this hasn't been seen in 3 months, okay to close?
Yes we fixed it with


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

PodInterface should have Watch method behaving in the same way as ServiceInterface.Watch does. Use case: a monitoring system (namely Prometheus) expects to get the list of monitored servers performing a DNS-SD query. Special service (similar to kube2sky shipped with Kubernetes) mirrors status of running Pods into etcd, which is in turn monitored by SkyDNS server and exported as DNS-SD. Schematically the data flow looks like this: Kubernetes -kube2sky -etcd -SkyDNS -Prometheus. The biggest challenge is to deliver list of all running pods from Kubernetes to kube2sky. Currently kube2sky can only poll Kubernetes with PodInterface.List() each N seconds, which is not efficient. Having a Watch method would be much better.
cc Do you know why we do not support watch?
Every object should support Watch.
Watching status doesn't currently work -- that's .
I agree with OP. This is just an accident of history; some boilerplate in the client isn't completely filled out. As Brian mentions, not every field is currently watchable but that's being addressed elsewhere.
This is relevant to your interests potentially w/ your work on
This is a pretty simple change if someone wants to get their feet wet. :)
I'll work on it.


Question: fi fi <del> if [[ -n "${1-}" ]]; then cover_report_dir="" if [[ -n "${KUBE_COVER}" ]]; then cover_report_dir="/tmp/k8s_coverage/$(kube::util::sortable_date)" kube::log::status "Saving coverage output in '${cover_report_dir}'" fi for arg; do trap 'exit 1' SIGINT pkg=${KUBE_GO_PACKAGE}/${arg} cover_params=() if [[ -n "${KUBE_COVER}" ]]; then mkdir -p "${cover_report_dir}/${arg}" cover_params=(${KUBE_COVER} -coverprofile="${cover_report_dir}/${arg}/coverage.out") fi </del> <ins> # If we're not collecting coverage, run all requested tests with one 'go test' # command, which is much faster. if [[ ! ${KUBE_COVER} =~ ^[yY]$ ]]; then kube::log::status "Running unit tests without code coverage" go test "${goflags[@]:+${goflags[@]}}"  ${KUBE_RACE} ${KUBE_TIMEOUT} "${@+${@/#/${KUBE_GO_PACKAGE}/}}" exit 0 fi # Create coverage report directories. cover_report_dir="/tmp/k8s_coverage/$(kube::util::sortable_date)" cover_profile="coverage.out"  # Name for each individual coverage profile kube::log::status "Saving coverage output in '${cover_report_dir}'" mkdir -p "${@+${@/#/${cover_report_dir}/}}" # Run all specified tests, collecting coverage results. Go currently doesn't # support collecting coverage across multiple packages at once, so we must issue # separate 'go test' commands for each package and then combine at the end. # To speed things up considerably, we can at least use xargs -P to run multiple # 'go test' commands at once. printf "%sn" "${@}" | xargs -I{} -n1 -P${KUBE_COVERPROCS}  </ins> go test "${goflags[@]:+${goflags[@]}}"  ${KUBE_RACE}  ${KUBE_TIMEOUT}  <ins> -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/{}/${cover_profile}"  </ins> "${cover_params[@]+${cover_params[@]}}"  <del> "${pkg}" done exit 0 </del> <ins> "${KUBE_GO_PACKAGE}/{}" combined_cover_profile="${cover_report_dir}/combined-coverage.out" { # The combined coverage profile needs to start with a line indicating which # coverage mode was used (set, count, or atomic). This line is included in # each of the coverage profiles generated when running 'go test -cover', but # we strip these lines out when combining so that there's only one. echo "mode: ${KUBE_COVERMODE}" # Include all coverage reach data in the combined profile, but exclude the # 'mode' lines, as there should be only one. find "${cover_report_dir}" -name "${cover_profile}" | xargs -r grep -h -v "^mode:" || true } >"${combined_cover_profile}" coverage_html_file="${cover_report_dir}/combined-coverage.html" go tool cover -html="${combined_cover_profile}" -o="${coverage_html_file}" kube::log::status "Combined coverage report: ${coverage_html_file}" if [[ -x "${KUBE_GOVERALLS_BIN}" ]]; then ${KUBE_GOVERALLS_BIN} -coverprofile="${combined_cover_profile}" || true </ins> fi <del> kube::test::find_pkgs | xargs go test "${goflags[@]:+${goflags[@]}}"  ${KUBE_RACE}  ${KUBE_TIMEOUT}  ${KUBE_COVER} </del>

Our test- script invokes 'go test' serially for each pkg - currently taking 2 minutes on my laptop. Running a single 'go test' for all pkgs cuts the time down to 45 seconds. Even running with -race is only 1:15. As far as I can tell, the only time we HAVE to run serially is when coverage is on. Does anyone have time to fix this?
I'd created on Friday to fix this, though it's not yet been reviewed.
I cobbled together but I'll review yours since it is probably better :) , Jeff Grafton wrote:


Question: "containers": [{ "name": "slave", "image": "brendanburns/redis-slave", <ins> "cpu": 200, </ins> "ports": [{"containerPort": 6379, "hostPort": 6380}] }] }

HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


Question: - name: SchedulingPreferredAntiAffinityWithNSSelector defaultPodTemplatePath: config/pod-preferred-anti-affinity-ns-selector.yaml <del> featureGates: PodAffinityNamespaceSelector: true </del> workloadTemplate: - opcode: createNodes countParam: $initNodes

/sig scheduling in the job is failing. success run Due to this failure, we cannot see any scheduler metrics results from the perf-dash. master
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind bug /kind flake /kind failing-test
/assign /remove-kind flake
will fixed by


Question: "//staging/src/k8s.io/component-base/metrics/...", "//test/e2e_node", "//test/integration/apiserver/flowcontrol", <ins> "//test/integration/metrics", </ins> "//vendor/...", ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While I was watching grafana for a single run, I noticed that GET requests are reported as LIST requests, e.g. grafana shows ~1K QPS for LIST of pods ! while kube-apiserver's logs show: (there are no pod lists that time). I'm not sure if this is intended change to metrics reporting (kind of unexpected to be honest), but this definitely affects way our scalability tests validate latency metrics: We do not validate GET API call latency at all Validation for LIST requests is also affected, as usually GET API calls are way cheaper than LIST requests, so grouping GET-LISTs and LIST-LISTs together lowers p99 for LISTs. Also perf-dash shows that GET requests simply disappeared at some point: http://perf- ! What you expected to happen: GET requests should be reported with verb=GET in apiserverrequest_total metric. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:

FYI
I managed to narrow the issue by bisecting the runs of which runs load tests using master k/k every 1 hour. This is the last test run that had GETs reported correctly: This is the first test run that has no GETs reported: Looking at the k/k repo commits' hashes inside artifacts in both runs and at what happened between each of them it looks as if might be the culprit.
From the former run's artifacts: This is (among others) what we miss in the latter run - this is understandable from the point of view of this if condition: Why did the reported GETs have as the scope instead of in the first place, however??
It seems that our handing is incorrect: vs Or my understanding of it was incorrect.

It seems that the semantic is: cluster - for cluster-scope requests namespace - for namespace-scoped requests request - for subresources I think the assumption in the monitoring path was: cluster - for cluster-scope requests namespace - for namespace-scope requests requests - for requests touching a single resource (not necessary subresource) I'm not 100% sure which semantic we should use - based on the finding we should fix one or the other place...
cc There should probably be a unit test mapping URL paths to the expected label values.
is opened to fix that.
/triage accepted


Question: "UserDefinedMetrics": gomega.BeEmpty(), }) } <ins> expectedPageFaultsUpperBound := 1000000 </ins> expectedMajorPageFaultsUpperBound := 10 if IsCgroup2UnifiedMode() { <del> expectedMajorPageFaultsUpperBound = 1000 </del> <ins> // On cgroupv2 these stats are recursive, so make sure they are at least like the value set // above for the container. expectedPageFaultsUpperBound = 1e9 expectedMajorPageFaultsUpperBound = 100000 </ins> } podsContExpectations := sysContExpectations().(*gstruct.FieldsMatcher)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-cos-cgroupv2-containerd-node-e2e <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --<!-- Please see the deflaking doc () for more guidance! --/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### Which jobs are flaking: links to appreciated links to specific failures in spyglass appreciated </details
On cgroupv2, it looks like like the number of for cgroup is sometimes higher than is currently expected in the test. We currently expect this however, in a few runs of the test we've seen We may need to adjust this value similar as was done in for
/cc
yes, good idea. I think we need to adjust the PageFaults test as well. I've opened a PR:
/assign /triage accepted /area test /priority important-soon


Question: fmt.Sprintf("sequence No: %v->%v", rs.Status.ObservedGeneration, newStatus.ObservedGeneration)) rs.Status = newStatus <del> _, updateErr = c.UpdateStatus(rs) if updateErr == nil || i >= statusUpdateRetries { return updateErr </del> <ins> updatedRS, updateErr = c.UpdateStatus(rs) if updateErr == nil { return updatedRS, nil } // Stop retrying if we exceed statusUpdateRetries - the replicaSet will be requeued with a rate limit. if i >= statusUpdateRetries { break </ins> } // Update the ReplicaSet with the latest resource version for the next poll if rs, getErr = c.Get(rs.Name, metav1.GetOptions{}); getErr != nil { // If the GET fails we can't trust status.Replicas anymore. This error // is bound to be more interesting than the update failure. <del> return getErr </del> <ins> return nil, getErr </ins> } } <ins> return nil, updateErr </ins> } func calculateStatus(rs *extensions.ReplicaSet, filteredPods []*v1.Pod, manageReplicasErr error) extensions.ReplicaSetStatus {

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: core.NewCreateAction(schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, metav1.NamespaceDefault, createdTokenSecret()), // Retry 1 core.NewGetAction(schema.GroupVersionResource{Version: "v1", Resource: "serviceaccounts"}, metav1.NamespaceDefault, "default"), <del> core.NewCreateAction(schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, metav1.NamespaceDefault, namedCreatedTokenSecret("default-token-x50vb")), </del> <ins> core.NewCreateAction(schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, metav1.NamespaceDefault, namedCreatedTokenSecret("default-token-jk9rt")), </ins> // Retry 2 core.NewGetAction(schema.GroupVersionResource{Version: "v1", Resource: "serviceaccounts"}, metav1.NamespaceDefault, "default"), <del> core.NewCreateAction(schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, metav1.NamespaceDefault, namedCreatedTokenSecret("default-token-scq98")), </del> <ins> core.NewCreateAction(schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, metav1.NamespaceDefault, namedCreatedTokenSecret("default-token-684pg")), </ins> }, }, "new serviceaccount with missing secrets": {

/kind feature What happened: In playing with the nginx ingress controller deployment, I noticed the following pod name: nginx-ingress-controller--c0ck5 I figure we should possibly put something in place to prevent accidental l33t profanity ;-). What you expected to happen: The following pod name: nginx-ingress-controller--r0ck5
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
My first github (k8s) issue. I've no idea which SIG would cover this topic. /sig product-management
/sig api-machinery
Previously we removed vowels in . We can 3xt3nd the idea with 0, 1 (can stand for ) and 3. would you like to send a PR? If not, I can work on this.
/assign jamesawebb1
Reassign to me if you do not want to contribute a PR.


Question: "sync" "time" <ins> "github.com/golang/glog" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/diff" )

/kind bug What happened: Start a cluster using ./hack/local- Now just monitor the memory on kube-apiserver and kube-controller-manager. Without doing anything else note the memory on each process has grown by 500 MB each day. Tried this for 5 days and saw this steady growth. What you expected to happen: Memory to remain relatively flat after some initial growth. How to reproduce it (as minimally and precisely as possible): Run ./hack/local- and wait. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10 Cloud provider or hardware configuration: local OS (e.g. from /etc/os-release): Debian GNU/Linux Kernel (e.g. ): Debian 4.9.65-3+deb9u2 Install tools: Others:
/sig api-machinery
/milestone v1.10
/priority important-soon
!
Copying some of what was discussed in Slack - we need to redo a test with set when running . This disables the cache mutation detector, which holds on to "all the things" .
What does the chart mean? yeah determined that most of it is from the mutation detector (it really makes me nervous to have test code linked in to the production binaries).
It was a differential pprof memory chart of hack/local-cluster- But as noted, we have to run it without mutation detection. I did that for an hour yesterday and did not notice much memory leaking.
After the suggested fix (turn of mutation detection), there was no noticeable increase on controller manager memory and the api-server memory increase was down to 8 MB per day. (on my setup)
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 2 days, the issue will be moved out of the v1.10 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
closing this to remove it from the release radar. Feel free to reopen if you think we still have an issue.


Question: s.lock.Lock() defer s.lock.Unlock() s.writeCalled = true <del> s.dataWritten = make([]byte, len(p)) copy(s.dataWritten, p) return len(p), io.EOF </del> <ins> s.dataWritten = append(s.dataWritten, p...) // Indicate the stream accepted all the data, and can accept more (no err) // Returning an EOF here will cause the port forwarder to immediately terminate, which is incorrect, in case someone writes more data return len(p), nil </ins> } func (s *fakeUpgradeStream) Close() error {

ok 1.520s coverage: 25.5% of statements I0414 23:19:35. ] Forwarding from 5000 -5000 W0414 23:19:35. ] Unable to listen on port {5000 5000}: listen tcp 127.0.0.1:5000: bind: address already in use I0414 23:19:35. ] Forwarding from 6000 -6000 I0414 23:19:35. ] Handling connection for 5000 --- FAIL: TestForwardPorts (0.22s) 2: expected to read '1234', got '' FAIL coverage: 83.3% of statements FAIL 0.677s
It looks like the crux is this: ] Unable to listen on port {5000 5000}: listen tcp 127.0.0.1:5000: bind: address already in use listen often needs a retry after a recent close, to account for TIME_WAIT state of the port. But that's pure speculation on my part. We need to repro and fix.
http://hea- is perhaps a better link.
I'll have a go at it in the next few days if nobody else picks it up before me.
There were two issues... one was multiple test cases reusing the same port and deferring the close until the end of the function, not the end of the loop... that was fixed in The other issue was with the fake stream returning an EOF on a Write(), fixed in


Question: } } path = strings.Replace(n.itemPath, "{name}", name, 1) <del> if !n.allNamespaces { path = strings.Replace(path, "{"+n.scope.ParamName()+"}", namespace, 1) } </del> <ins> path = strings.Replace(path, "{"+n.scope.ParamName()+"}", namespace, 1) </ins> return path, "", nil }

Accessing entities without namespaces in the url makes the selflink to look corrupted (has namespace missing): Example - invoking: http://localhost:8080/api/v1beta3/services/ returns entities with this selflink (note the part after namespaces) "selfLink": "/api/v1beta3/namespaces/%7Bnamespaces%7D/services/guestbook"
I thought accessing namespaced objects without the namespace was just disallowed in v1beta3. ?
Oh, probably list is allowed for components that need to watch all objects across all namespaces, such as kube-proxy. In that case, the selfLink looks broken, yes.
- right. I will look into the self link this afternoon to see what's going on. Self assigning.
The api has to allow fetching all objects cross namespaces without specifying namespace, otherwise it will cause multiple requests to the api - per namespace, big overhead for clients.


Question: if j.beginRange > 0 { j.beginRange-- j.inRange++ <del> for _, value := range results { </del> <ins> if len(results) > 0 { for _, value := range results { j.parser.Root.Nodes = nodes[i+1:] nextResults, err := j.FindResults(value.Interface()) if err != nil { return nil, err } fullResult = append(fullResult, nextResults...) } } else { // If the range has no results, we still need to process the nodes within the range // so the position will advance to the end node </ins> j.parser.Root.Nodes = nodes[i+1:] <del> nextResults, err := j.FindResults(value.Interface()) </del> <ins> _, err := j.FindResults(nil) </ins> if err != nil { return nil, err } <del> fullResult = append(fullResult, nextResults...) </del> } j.inRange--

What happened: throws errors if no pods are found in v1.19.3, but returns success with empty response in v1.16.8 What you expected to happen: If there are no pods that return, no error should be thrown as the range should evaluate to "no op". How to reproduce it (as minimally and precisely as possible): Adapt the example from to get the command . Run this when there are pods in default namespace - all is well. Run this when there are no pods in default namespace and you get the error: Anything else we need to know?: This breaks all kinds of code that relies on jsonpath's earlier behaviour. I found an old report () that seems appropriate, but that is long since merged and closed. Environment: Kubernetes version (use ): 1.19.3 Cloud provider or hardware configuration: x86-64 OS (e.g: ): RedHat Enterprise Linux 7.6 (Maipo) Kernel (e.g. ): Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
/sig cli
/cc
/sig api-machinery
/assign cc /triage accepted


Question: EnableV1Beta3:          v1beta3, MasterServiceNamespace: s.MasterServiceNamespace, ClusterName:            s.ClusterName, <ins> SyncPodStatus:          s.SyncPodStatus, </ins> } m := master.New(config)

Split off as a smaller more specific work item. Apiserver should have a flag that makes it not attempt to GET from kubelets. This would be used in conjunction with . After we have converted the top N Getting-started-guides to the new style, we can rip out the kubelet client from the apiserver altogether.
This is tightly related to so I'll take it.


Question: <ins> # Admission Controllers ## What are they? An admission control plug-in is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object.  The plug-in code is in the API server process and must be compiled into the binary in order to be used at this time. Each admission control plug-in is run in sequence before a request is accepted into the cluster.  If any of the plug-ins in the sequence reject the request, the entire request is rejected immediately and an error is returned to the end-user. Admission control plug-ins may mutate the incoming object in some cases to apply system configured defaults.  In addition, admission control plug-ins may mutate related resources as part of request processing to do things like increment quota usage. ## Why do I need them? Many advanced features in Kubernetes require an admission control plug-in to be enabled in order to properly support the feature.  As a result, a Kubernetes API server that is not properly configured with the right set of admission control plug-ins is an incomplete server and will not support all the features you expect. ## How do I turn on an admission control plug-in? The Kubernetes API server supports a flag, ```admission_control``` that takes a comma-delimited, ordered list of admission control choices to invoke prior to modifying objects in the cluster. ## What does each plug-in do? ### AlwaysAdmit This plug-in will accept all incoming requests made to the Kubernetes API server. ### AlwaysDeny This plug-in will reject all mutating requests made to the Kubernetes API server.  It's largely intended for testing purposes and is not recommended for usage in a real deployment. ### DenyExecOnPrivileged This plug-in will intercept all requests to exec a command in a pod if that pod has a privileged container. If your cluster supports privileged containers, and you want to restrict the ability of end-users to exec commands in those containers, we strongly encourage enabling this plug-in. ### ServiceAccount This plug-in limits admission of Pod creation requests based on the Pod's ```ServiceAccount```. 1. If the pod does not have a ```ServiceAccount```, it modifies the pod's ```ServiceAccount``` to "default". 2. It ensures that the ```ServiceAccount``` referenced by a pod exists. 3. If ```LimitSecretReferences``` is true, it rejects the pod if the pod references ```Secret``` objects which the pods ```ServiceAccount``` does not reference. 4. If the pod does not contain any ```ImagePullSecrets```, the ```ImagePullSecrets``` of the ```ServiceAccount``` are added to the pod. 5. If ```MountServiceAccountToken``` is true, it adds a ```VolumeMount``` with the pod's ```ServiceAccount``` API token secret to containers in the pod. We strongly recommend using this plug-in if you intend to make use of Kubernetes ```ServiceAccount``` objects. ### SecurityContextDeny This plug-in will deny any ```SecurityContext``` that defines options that were not available on the ```Container```. ### ResourceQuota This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the ```ResourceQuota``` object in a ```Namespace```.  If you are using ```ResourceQuota``` objects in your Kubernetes deployment, you MUST use this plug-in to enforce quota constraints. It is strongly encouraged that this plug-in is configured last in the sequence of admission control plug-ins.  This is so that quota is not prematurely incremented only for the request to be rejected later in admission control. ### LimitRanger This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the ```LimitRange``` object in a ```Namespace```.  If you are using ```LimitRange``` objects in your Kubernetes deployment, you MUST use this plug-in to enforce those constraints. ### NamespaceExists This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes ```Namespace``` and reject the request if the ```Namespace``` was not previously created.  We strongly recommend running this plug-in to ensure integrity of your data. ### NamespaceAutoProvision (deprecated) This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes ```Namespace``` and create a new ```Namespace``` if one did not already exist previously. We strongly recommend ```NamespaceExists``` over ```NamespaceAutoProvision```. ### NamespaceLifecycle This plug-in enforces that a ```Namespace``` that is undergoing termination cannot have new content created in it. A ```Namespace``` deletion kicks off a sequence of operations that remove all content (pods, services, etc.) in that namespace.  In order to enforce integrity of that process, we strongly recommend running this plug-in. Once ```NamespaceAutoProvision``` is deprecated, we anticipate ```NamespaceLifecycle``` and ```NamespaceExists``` will be merged into a single plug-in that enforces the life-cycle of a ```Namespace``` in Kubernetes. ## Is there a recommended set of plug-ins to use? Yes. For Kubernetes 1.0, we strongly recommend running the following set of admission control plug-ins: ```shell --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota ``` </ins>

As the title suggests, I can find no documentation on KUBEADMISSIONCONTROL (from the apiserver configuration), and I don't know golang well enough to determine the answer (yet) from the source code. Google search yields only 3 hits today, and all of them are copies of the configuration file itself with no commentary. As an admin, I'd be grateful for simple documentation on all the variables, their possible settings, and what those really mean. If that already exists, please identify where it exists. Though my formal issue is the documentation, the problem that leads me here is that my pods can't query the apiserver. Given the following, my connection is reset: Any assistance would be greatly appreciated, though my formal issue remains the lack of thorough documentation on the configuration. I'll continue learning golang in order to try to figure out what my problem may be.
I agree a document is needed, will look to put something together describing what each option does.
Thanks! :)
That being said, I recommend we scope this issue to just documentation being needed, and if you need assistance with your other topic, treat it as a separate issue.
Quite reasonable! If you think that documentation will be before the weekend, I'll wait until that's available and see if I can figure it out myself; otherwise, I'll make a separate issue for that.


Question: "flag" "fmt" "log" <ins> "math/rand" </ins> "net" "os" "strconv"

In Kubernetes 1.26, Kubernetes Enhancement Proposal, KEP-3352[1]: Aggregated Discovery, enhanced the discovery mechanism and supported aggregated discovery optimizing response for discovery api. To support this KEP, Changes were made to both API server and client to support aggregated discovery mechanism. This feature was available alpha in Kubernetes 1.26 behind a feature flag, . In Kubernetes 1.27, aggregated discovery endpoint was promoted to beta and enabled by default[2]. Here is a class of problems that ended up getting triggered. GOOD NEWS!! is that it is already fixed by in 1.27 and a cherry pick in 1.26 For example, With Prometheus adapter, we saw issues across repositories: One signature of this is following in the traceback:
Searching for that method signature: Currently shows the following:
Leaving this open for some time, just to serve as a point of coordination if needed. The advice/guidance here is for folks to update to newer versions of 1.26/1.27 client-go.
/sig api-machinery
Components that do generate custom metrics (grepping through sources!): etc..
Thanks, Any custom server, sending nil GVK could trigger this bug. So far have seen it through custom metrics via promethus-adapter.
Thanks for tracking this /cc /triage accepted
For those searching in the future, the impacted versions are: 26.0 - 0.26.3 The fixed versions are: 26.4+ As well as updating , make sure that your other dependencies do not pull in any of the affected versions.
/close i think that's about it for now..
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } if pointedModel == nil { <del> return fmt.Errorf("Requested resourse: %s doesn't exit", inModel) </del> <ins> return fmt.Errorf("Requested resource: %s doesn't exist", inModel) </ins> } if len(fieldsPath) == 0 {

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale