Question: pkg/controller/volume/attachdetach/config/v1alpha1 pkg/controller/volume/attachdetach/statusupdater pkg/controller/volume/attachdetach/testing <del> pkg/controller/volume/events </del> pkg/controller/volume/expand pkg/controller/volume/persistentvolume pkg/controller/volume/persistentvolume/config/v1alpha1

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func (tc *legacyTestCase) runTest(t *testing.T) { testClient, testScaleClient := tc.prepareTestClient(t) metricsClient := metrics.NewHeapsterMetricsClient(testClient, metrics.DefaultHeapsterNamespace, metrics.DefaultHeapsterScheme, metrics.DefaultHeapsterService, metrics.DefaultHeapsterPort) eventClient := &fake.Clientset{} eventClient.AddReactor("*", "events", func(action core.Action) (handled bool, ret runtime.Object, err error) { tc.Lock() defer tc.Unlock() <ins> if tc.finished { return true, &v1.Event{}, nil } </ins> obj := action.(core.CreateAction).GetObject().(*v1.Event) if tc.verifyEvents { switch obj.Reason {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: g := NewGraph() g.destinationEdgeThreshold = 3 <del> a := NewAuthorizer(g, nil, nil).(*NodeAuthorizer) </del> <ins> a := NewAuthorizer(g, nil, nil) </ins> addPod := func(podNumber, nodeNumber int) { t.Helper()

What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


Question: container.Args[e] = strings.Replace(container.Args[e], "/"+o.OldDriverName+"/", "/"+o.NewDriverName+"/", 1) } } <ins> // Modify --kubelet-registration-path. for e := range container.Args { container.Args[e] = substKubeletRootDir(container.Args[e]) } for e := range container.VolumeMounts { container.VolumeMounts[e].MountPath = substKubeletRootDir(container.VolumeMounts[e].MountPath) } </ins> // Overwrite driver name resp. provider name // by appending a parameter with the right // value.

What happened: The CSI hostpath driver used in the e2e tests expects the root directory of nodes to be set to , , however nodes can be configured to use different paths. The tests should find out the root directory from kubelet to allow for this. Several test failures related to CSI Volumes: What you expected to happen: Test should be expected to pass. How to reproduce it (as minimally and precisely as possible): Configure kubelet on cluster nodes to use the non-default path. Run the storage-csi e2e tests. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig storage /remove-sig testing This is a SIG storage specific issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How? I've asked about that before and the conclusion was that there is no portable way to do that (i.e. no official API). It seems to fall onto cluster admins to configure CSI driver deployments so that this particular aspect matches the cluster. For the tests that means that there has to be a parameter for the kubelet data directory that users of then have to set.
We could add and respect a flag of course then all nodes have to be the same at least. Reminder though
That's indeed another complication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Tracked for csi-driver-host-path in There's a bigger issue about both mock and hostpath driver in which includes "support non-default kubelet path".
The current hostpath driver in k/k testing supports non-default paths, but the suite doesn't use that yet.

/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // allows to pass an offset that divides the allocation bitmap in two blocks. // The first block of values will not be used for random value assigned by the AllocateNext() // method until the second block of values has been exhausted. <ins> // The offset value must be always smaller than the bitmap size. </ins> func NewAllocationMapWithOffset(max int, rangeSpec string, offset int) *AllocationBitmap { a := AllocationBitmap{ strategy: randomScanStrategyWithOffset{

kube-apiserver observed a panic: kube-apiserver should not panic Not sure yet -- will add more details as I investigate. No response v1.25.4 N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /cc
max is greater than offset?
/assign


Question: "io/ioutil" "log" "math/rand" <del> "net" </del> "net/http" <ins> "net/url" </ins> "os" "sync" "time" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" )

It's failing on both GCE and GKE-CI. It appears to have broken somewhere around build number 2509/2510 which includes a likely candidate PR (). /cc
I was about to debug this one. I'll take it.
Root cause is almost definitely one of these: Summary secrets example pod (details) the permissions on python files. (details) the implementation of DeepCopy to use reflection instead of Gob encode/decode. (details) networking test (details) back to executable. (details) protected docker processes (-900) (details) proxySocket out of to (details) Commit by Fix secrets example pod The file was modified Commit by Fixing the permissions on python files. The file was modified cluster/juju/charms/trusty/kubernetes- The file was modified cluster/juju/return-node- The file was modified cluster/juju/charms/trusty/kubernetes- Commit by Changing the implementation of DeepCopy to use reflection instead of Gob encode/decode. The file was The file was modified The file was modified Commit by Fix networking test The file was modified contrib/for-tests/network- The file was modified contrib/for-tests/network- The file was modified The file was modified contrib/for-tests/network-tester/Makefile Commit by Changing back to executable. The file was modified Commit by OOM protected docker processes (-900) The file was modified The file was modified The file was modified Commit by Move proxySocket out of to Moves proxySocket out of to new in proxy package in order to start separating proxy logic and implementation and make proxier more manageable to review. The file was modified The file was
Pretty sure it's commit , so rolling back now. cc .
The above rollback solved that immediate problem, but now the test, while proceeding healthily, is timing out. More debugging or optimization required: Networking should function for intra-pod communication ..... [irrelevant stuff deleted] ..... INFO: Proxy status call returned in 4.861944ms INFO: Attempt 35/60: test still running INFO: About to make a proxy status call Build timed out (after 90 minutes). Marking the build as aborted. Build timed out (after 90 minutes). Marking the build as failed. Ran 40 of 46 Specs in 5069.444 seconds FAIL!

-- 18 Passed 5 Skipped Build was aborted Recording test results
I'm not entirely sure why, but the test is running normally again, completing consistently in 13 seconds. Closing this issue for now. !


Question: // ListByResourceGroup indicates an expected call of ListByResourceGroup func (mr *MockInterfaceMockRecorder) ListByResourceGroup(ctx, resourceGroupName interface{}) *gomock.Call { mr.mock.ctrl.T.Helper() <del> return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListByResourceGroup", reflect.TypeOf((*MockInterface)(nil).Delete), ctx, resourceGroupName) </del> <ins> return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ListByResourceGroup", reflect.TypeOf((*MockInterface)(nil).ListByResourceGroup), ctx, resourceGroupName) </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ListByResourceGroup method in the azure diskClient mock makes a call to Delete What you expected to happen: ListByResourceGroup method in the azure diskClient mock should call ListByResourceGroup How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cloud-provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/area provider/azure
fixed by /close
Closing this issue. <detailsIn response to : fixed by /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/google/cel-go/common/types/ref" "github.com/google/cel-go/common/types/traits" <ins> "k8s.io/kube-openapi/pkg/validation/strfmt" </ins> "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apiserver/pkg/cel" <del> "k8s.io/kube-openapi/pkg/validation/strfmt" </del> ) // UnstructuredToVal converts a Kubernetes unstructured data element to a CEL Val.

Added this test to : this resulted in: The test should pass. Add the above test No response All versions where CEL is used. All Doesn't matter No response No response No response_
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Opened to discuss the expected behavior in cel-go for this.


Question: activeEndpoints := make(map[proxy.ServicePortName]bool) // use a map as a set staleConnections := make(map[endpointServicePair]bool) svcPortToInfoMap := make(map[proxy.ServicePortName][]hostPortInfo) <ins> newEndpointsMap := make(map[proxy.ServicePortName][]*endpointsInfo) </ins> // Update endpoints for services. for i := range allEndpoints {

3 node Kubernetes v1.2.4 cluster with HA scheduler and controller-manager have no additional POD running aside from those in the namespaceruns twice per second. Increasing verbosity shows that this is triggered by Stopping all the schedulers and controller-managers, then restarting only one instance of each without stops from running that often.
That seems to be related to
We should try to filter updates to the proxy when the only change is an annotation that it's not watching. This seems to be an unintended consequence of leveraging endpoints.
I'm going to use this one as the root issue.
Do the frequent restores cause any problems?
Performance wise we've had several issues... they can be computationally expensive the larger the number of routes you add.
It's fairly anecdotal, but during the brief time these restores are running, they seem to consume a fair amount of CPU. Especially on an already very heavily-loaded node, this is undesirable.
We should be able to calculate when a restore is a noop and not run the restore. is anyone currently working in this area?
To really know whether an iptables sync is needed you have to do all the work except actually updating it. Working around this in iptables seems like a bad answer - this is delivering 2 events per second per node. Is that really necessary?
It's one update every two seconds by default and it's configurable. By increasing that period, you make a tradeoff with responsiveness of cluster failover. Conceivably updates in large clusters are much more frequent.
maybe Endpoints was the wrong object to watch? , Mike Danese wrote:
From an API perspective, annotating Endpoints, a collection of pods that implements a service, with which endpoint in the collection is the current master makes sense. I don't think anywhere else makes as much sense except for possibly a seperate resource. It's unfortunate that the kube-proxy cannot handle frequent updates to fields that should be noop on these objects. If we can't fix that generally then it shouldn't be hard to special case a fix for the leader election annotation. I'd still like someone to quantify what the effects of these frequent updates are e.g.: frequency/duration/magnitude of utilization spikes whether that utilization is spent traversing the state data structure in process or spent in fork execs This info would be very helpful in priortizing this.

There are only a few objects that are watched by every single node in the cluster. I know it makes sense to be there, but it really is sort of a pessimal choice. , Mike Danese wrote:
We were in favor of component registration, but that seems to have died.
! As you can see from the output of here (rows highlighted in pink), / are using a non-trivial amount of resources over a given 10 second interval; ~1.5 seconds of CPU time and 42 processes launched. You said you wanted to know what was happening internally, but given the number of processes – there were 14 invocations of and 28 of – it's difficult to know precisely what I should . In any case, for something that is essentially a no-op, this activity seems excessive and undesirable.
So we have a BZ open against too many iptables refreshes on heavily loaded clusters but all the comments are private: . The distillation is it's eating time in the kernel (kernel-commit [net] netfilter: xtables: speed up jump target validation" that speeds up iptables operations somewhat.) However, there are still issues with incessant refreshes + large number of rules.
If someone wants to make the kube-proxy logic smarter in the face of no-op changes, I'm happy to entertain that patch, but I am more concerned thatthis is broadcasting events to every node every couple seconds. , Timothy St. Clair
imo we need to lock on a different object, at 5-10k nodes this is an untenable level of overhead.
The anecdote indicates that the immediate perf issue is caused by iptables-restore. In small clusters we can alleviate this by not restoring noops. I'd like to see the frequency of updates in large clusters in regular state (e.g. pods dieing, deployments in progress). Annotation updates may be a small portion of total endpoints updates. If endpoints updates are happening anyway and they are batched, then moving off of endpoints may not be useful. If these updates are not noops then optimizing out noop iptables-restores will also not be useful. I'd be happy with moving to another resource if we identify a better place for this. I'm also happy to revisit the default config for the leaderelection client.

I'd be unhappy with a change that diminishes the clarity of the API on an assumption that it will improve performance. It's possible that the tradeoff is worth it but we need to validate/quantify that assumption.
They are batched, but the watch updates will still occur across the cluster. So the more components we have annotating different endpoints the larger the fan-out. Just the notion of a single annotation could cause 10k watch updates for actors who don't care and then have to filter gives me pause. /cc
given that 3rd party api is now in place, would you have any objections if I were to make a pull to allow locking there?
nope, sounds fine.
Exiting crazy release+replanning, part 1 is available for review, onto part 2.
Third party resources don't have validation or conversions across versions. You should probably consider making this a first class API object to gain those benefits. Otherwise things may get really strange across version migrations.
So the only other object that seems appropriate might be component-status. thoughts?
I definitely think that ComponentStatus is better than Endpoints.
k, making change..
SGTM, but IIRC, component status is not backed by etcd?
I just found that out a second ago :-(
So were kind of in a bit of a conundrum. 3rd party doesn't work, and component status isn't stored in etcd. So I see 2 options: the logic for querying componentstatus into something like the node controller and reflect that data etcd. to revisit a scaled down version of component registration to address add-ons in the kubeadm world? Thoughts? /cc
When last I looked at it, the component stuff was heading in a direction to be a convention on top of services/endpoints. That seems right to me -- it is a way to move away from a special type of endpoint that was rushed in a while ago. That doesn't solve the problem of updating leader information. My initial read (without studying this deeply) is that spamming updates to the endpoints object is the wrong way to do this. Can we only update the endpoints object once a new leader is elected and do the spammy stuff elsewhere? (I think that is what the discussion about third party resources used for this is).

At the end of the day, is this something that kubernetes should be involved in? If we had a lock server that everyone could talk to (with a security identity and quotas) then that could be used for leader election. Do we want to turn the api server into a generic lock server too?
Only for integrated components, or add-ons, of kubernetes. If we're going to self host the control plane, it's now an implicit requirement if you want HA, otherwise you're adding another etcd which would need it's own setup+config.
What would be really nice is if we could filter a watch based on a field path and only recieve updates if those fields are updated. We need some place to store the lease record. Every resource in kubernetes is a already a lock.
I don't want this to fall off the rails. So it sounds like Option 2 is thorny, but definitely an issue to address as add-ons/components grow. We need a secure reliable mechanism to register with the system and stating leader information is a pretty natural thing to do with such a facility. In the long run, I still believe it is the correct solution, but I doubt we are going to get there soon. That being said the issue presented here is an immediate concern and I don't see any major down sides to Option 1 listed above, but I'd need to do it in a couple of parts.
xref (the component status PR ) which is lost forever because appearantly chokes on it?
I only just realized this was happening after I think the real cost of kubeproxy restoring rules multiple times a second is increased iptables lock contention (and cpu). It might stall concurrent kubelet iptables modifications for some smallish period. I'm inclined to not put in a spot fix for kubeproxy, but I want to make sure it isn't an issue, ?
Is it a spot fix to not call restore if there is no diff in the rules? How hard is it to calculate this diff? That doesn't resolve this issue but it seems like it would mitigate one of the bad symptoms.
I think it should be easy. Definitely should make 1.5.

If no ones impacted by this "hotlooping", cherrypick into 1.4.X will depend on exactly how easy the fix is, if people are impacted, we should fix and cherrypick regardless. I'll need to poke around a bit.
I don't agree with the fix suggested by Basically the problem is that every "Endpoints" object is watched by all KubeProxy in the cluster. Which means, that if we are renewing the lease every 2s (which IIUC is the current case), and we have 2000-node clusters, that means that 2000 KubeProxy components are processing that Endpoint update every 2s. No matter if we will be updating iptables in that case or not, this is clearly a waste of time. For leader election we should use an object that components that run on nodes don't listen to - this in my opinion is pretty important requirement in this dicussion.
So I need "buy-in" on the proposed fix around pushing component status into a etcd-backed resource which it currently isn't. Are there any oppositions to this?
If you want to make it so kubeproxy doesn't get a watch event that it needs to ignore, less work for me :) I was proposing something in the here and now, given that this is happening in live 1.4 clusters. I'm guessing component status is not landing in time for 1.4.2.
yeah - I'm not sure if we even cherrypick it at all. So if we want to fix it in 1.4 branch (do you think we want - I don't know how important it is), then probably what you described should be a temporary solution. - regarding buy-in, I don't have any strong objections about it. Though may have some. WDYT?
I haven't seen this until now. What component is using the endpoint objects to do locking?
I would prefer to see us provide pods a way to present any information currently in componentstatus in their own pod status, and get rid of componentstatus entirely. It's a service we should provide for all components, not just system components.
Short term, assuming the locking behavior is legit, either kube-proxy needs to filter out fields it doesn't care about, or we need to get apiserver to filter out such changes (better but much more complex).

If kube-proxy is relying on the system to minimize the number of update messages it gets, that's a mistaken assumption in kube-proxy.
here's another user for your alternate view proposal.
Controller manager and scheduler have done this since 1.2.
That should affect a vanishingly small percentage of endpoints, so not much point to doing the extra work to fix it in apiserver. IMO this is just a plain old kube-proxy bug. , Mike Danese wrote:
kube-proxy can be smarter for sure, but this still broadcasts a useless change to every node every 2 seconds. It really should be a different object that is less likely to be actually used. IMO , Daniel Smith wrote:
+1000 - this is exactly what I was trying to express in
+1000 to and , The fan-out at large scale is simply untenable. So on the scale of a 10k node cluster where we had say a minimum of 2 components being updated every 2 seconds that is 10k x (60/2)*2 = 600,000 broadcasts in a minute even with filtering on the proxy side. I for one am a fan of open transparent communication, but this may be a bit too much ;-).
Ah, now I see what you're talking about. I agree it sounds bad, but is it a big percentage of the endpoints traffic at that scale? This costs 10k broadcasts per second. Assume there's 50 pods per node. Assume each pod shows in one service. Assume a pod changes status once every thousand seconds (most pods won't change often, but pods get started/stopped and crashloop often enough to raise the average). 50  10k  .001 = 500 changes per second, makes 5,000,000 broadcasts per second. So the number you are focusing on is like 1% of the number of endpoint changes at that scale... Maybe my numbers are off above, but are they off by two orders of magnitude? I doubt it. So really, you are running into a scaling problem that's fundamental to endpoints, and none of the above solutions will really address it.
I think there's actually a very old issue about endpoints not scaling, probably knows off the top of his head.

(Note that I don't object to changing scheduler and controller manager to use a different object for locking--endpoints was a weird choice IMO--but it is only adding a few feet onto the end of your runway.)
DO NOT persist componentstatuses in etcd. That API is a disaster. See for discussion of its future directions. I don't think endpoints scalability has its own issue, though it has been mentioned in a bunch of places: Pagination is
Ah, I may be misremembering IRL conversations as github issues... , Brian Grant wrote:
Endpoints as an API object has scalability problems (needs some form of pagination) and kube-proxy should batch updates (and filter no-ops). I am 99% sure both of these have open issues. , Daniel Smith wrote:
I don't see how pagination addresses the fan-out problem, or any problem. Even in a 10k node cluster there's only 500k - 1m pods total, and those will be merged into fewer endpoints objects, maybe 2x to 1000x fewer, depends on how big the services are. I think adding a low-cost endpoints view using the mechanism Clayton proposed is not enough; we can filter out metadata changes, ordering changes, and inactive endpoint changes, which kube-proxy doesn't care about (right?). I think that, at best, that filtering will only be 2-3x reduction, which is not enough. So intelligent update batching will be required, or we need a different distribution mechanism. (kube-proxy binaries could do some sort of p2p endpoint distribution, for example.) , Tim Hockin wrote:
(The right kind of update batching would require extensive changes to our watching model, in case it wasn't clear.)
Pagination is about the 1M etcd limit - at ~16 bytes to store an IPv4 address in JSON we will overflow that object after 32k-64k endpoints in the best case, potentially much less in the worst case. I know that feels like a lot, but I'd rather not get surprised by this one. This is not unique to endpoints. , Daniel Smith wrote:
If you have lots of services then, yes. That said I don't think it's super urgent problem. - do you have the same limit for size of value in etcd v3?

Guys, I'm sorry to jump in here with little context, but we need some solution to this issue in the short term. This is seriously impacting the performance of our nodes in production, and from my perspective it would make sense to add a fix that addresses the immediate issue now (probably in kube-proxy), and then separately make the endpoints API better in the medium term. It would be a shame for this issue to sit for ages while we debate the proper solution. Does this make sense? I'm very happy to pitch in and write some code if necessary
Above volunteered to look into the kube-proxy performance issue. Thanks, Beeps.
How about a configmap?
I was waiting for someone to complain (). Can you clarify what exactly is the issue? () just cpu or are you running something else that needs the iptables lock? kube-proxy should already have a request: 100m, meaning that's what it gets if there's contention, and if there isn't it can take how much it needs.
forked, this issue can be about node local implications of spammy watch notifications for kube-proxy
That's not pagination, that's sharding the endpoint object, which may not be necessary in etcd3 on account of size. I'd sooner worry about write contention with that many endpoints. (Pagination is where apiserver serves a list request in multiple pages.)
I guess my point is more that if kube-proxy / iptables can't handle the 2 second updates without being a CPU hog, it's going to have major problems in even medium sized clusters. Is there something odd about OP's setup? Why aren't other people complaining? Why does kube-proxy update iptables as fast as it can, why not batch and update every 5 seconds or 15? Can it precompute something and then take a lock for less time?
Yeah I think there's kubeproxy optimization and unnecessary watch notifications from overloading an object broadcast cluster wide. That's why i forked the bug.
They're not paying close attention to node performance profiles...? Also, we do HA differently so it never jumped out in our profiles, except during bulk operations. (see comment below) Broadcasting is still an issue, but we've proposed a similar approach:
Yes, the issue is CPU performance. In my you can see the sort of CPU is consuming.

On heavily-loaded nodes, this penalty and the associated context switches are not insignificant. I'm not sure why others are not seeing this issue to be honest; we are running HA master, on AWS, with some quite heavily-loaded worker nodes. I don't know how common this is, or as points out, how much attention others are paying to their node performance. It's also possible that we are doing something peculiar, but I can't think what. If you want me to gather more data about the performance impact this has I can, but I think I can say with certainty that this is a real problem.
Hmm, in that comment it looks like you have nothing else on the node. If you're saying it's getting much more than its "request: 100m" even on a loaded node, that sounds like a different bug. Of course we should still fix it, I'm just wary of pointing fingers at something that has a relatively low request.
Agree, there's an open issue for proxy to batch updates. There's really no need for it ever to run more than once per second, and even that is probably too much. , Daniel Smith wrote:
FTR - I did try to to repro this and had kube-proxy writing out thousands of iptables rules as fast as it could, and I never saw a single iptables-restore more than 120ms or so, (numbers from memory) , Prashanth B wrote:
/cc - fyi...
Since nobody mentioned this yet: the sheer amount of logging is causing an issue in itself. Is it really necessary to write out iptables this verbose? Also in journalctl it all shows up in red suggesting it is sent to stderr which made me feel uncomfortable looking at this for the first time. I searched for errors, but there were none.
IIRC you said you had a 3.x something kernel, while all our customers are either using later kernels, or using a highly rebased network stack from later kernels. I was able to reproduce the issues with a 4.7 kernels on Fedora. I haven't tried recent Ubuntu. But in any case, iptables-restore latency is highly kernel dependent it seems.
What verbosity are you running? The normal setting is --v=2 and that does not log all the iptables state.

That is --v=3 , leo-baltus wrote:
It sounds more and more like a kernel regression , Tim Hockin wrote:
Ah, running with --v=4. I wasn't aware of this, this is a setup made by kubeadm. Sorry for the noise. Op 25/10/2016 om 11:19:28 -0700, schreef Tim Hockin:
So if the fan-out is the problem, perhaps we want publish/subscribe instead? A message queue?..
cross-linking . We are a bit behind on versions (still on 1.3) and at least there, there are real updates to the default/kubernetes endpoints object. These come from the apiservers each thinking they're the one and resetting the endpoints list. suggests that there is a flag that needs to be set to remedy that but there is no documentation?
I'm trying to troubleshoot networking issues in my cluster by watching iptables packet counts using . It looks like a symptom of this issue is that the packet counters are continuously being reset by reloading iptables. Also, the rule order also gets updated continuously which makes it difficult to focus on one rule. Am I correct in assuming this issue is the cause?
probably, yes. , ravishivt <:


Question: var _bintree = &bintree{nil, map[string]*bintree{ "translations": {nil, map[string]*bintree{ <ins> "OWNERS":     {translationsOwners, map[string]*bintree{}}, </ins> "extract.py": {translationsExtractPy, map[string]*bintree{}}, "kubectl": {nil, map[string]*bintree{ "OWNERS": {translationsKubectlOwners, map[string]*bintree{}},

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: By("updating a scale subresource") scale.ResourceVersion = "" //unconditionally update to 2 replicas scale.Spec.Replicas = 2 <del> err = c.AppsV1beta1().RESTClient().Put().AbsPath("/apis/apps/v1beta1").Namespace(ns).Resource("statefulsets").Name(ssName).SubResource("scale").Body(scale).Do().Into(scaleResult) </del> <ins> err = c.AppsV1beta2().RESTClient().Put().AbsPath("/apis/apps/v1beta2").Namespace(ns).Resource("statefulsets").Name(ssName).SubResource("scale").Body(scale).Do().Into(scaleResult) </ins> if err != nil { framework.Failf("Failed to put scale subresource: %v", err) }

What happened: There are no e2e and integration tests for statefulset scale subresource. What you expected to happen: (1) e2e test should exist BEFORE code freeze (2) integration test should exist AFTER 1.8 release branches cut Anything else we need to know?: Refer PR for more info. We should NOT remove existing e2e test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: } } <ins> if o.list { accessor, err := meta.Accessor(outputObj) if err != nil { return err } indent := "" if !singleItemImpliedResource { indent = " " gvks, _, err := unstructuredscheme.NewUnstructuredObjectTyper().ObjectKinds(info.Object) if err != nil { return err } fmt.Fprintf(o.ErrOut, "Listing annotations for %s.%s/%s:n", gvks[0].Kind, gvks[0].Group, info.Name) } for k, v := range accessor.GetAnnotations() { fmt.Fprintf(o.Out, "%s%s=%sn", indent, k, v) } return nil } </ins> return o.PrintObj(outputObj, o.Out) }) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : add --list option to kubectl annotate command Why is this needed: like kubectl label commandshow annotations quickly
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign


Question: // objects and subsequent deltas. // Run will exit when stopCh is closed. func (r *Reflector) Run(stopCh <-chan struct{}) { <del> klog.V(3).Infof("Starting reflector %v (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) wait.Until(func() { </del> <ins> klog.V(2).Infof("Starting reflector %s (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) wait.BackoffUntil(func() { </ins> if err := r.ListAndWatch(stopCh); err != nil { utilruntime.HandleError(err) } <del> }, r.period, stopCh) </del> <ins> }, r.backoffManager, true, stopCh) klog.V(2).Infof("Stopping reflector %s (%s) from %s", r.expectedTypeName, r.resyncPeriod, r.name) </ins> } var (

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We recently had a production issue where one of our large cluster (~3k nodes) has it's API server OOMKilled. This caused all kubelets' connection to master broken, and caused a global synchronization of kubelet's reconnecting to master. Due to the burst of API call (18K QPS, 7x than normal), API server went into even worse condition, and caused more replicas to get OOMKilled. Current reflector impl has fixed 1sec backoff before calling next , which shall be the root cause of such instability. What you expected to happen: reflectors to backoff with jitter How to reproduce it (as minimally and precisely as possible): a cluster with a few thousands of nodes api server and reboot it Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig api-machinery scalability
/cc


Question: <ins> # `kube-log-runner` (formerly known as go-runner) The `kube-log-runner` is a Go based binary that can run commands and redirect stdout/stderr etc. Why do we need this? - Some of our images like kube-apiserver used bash output redirection for collecting logs, so we were not able to switch to distroless images directly for these images. The klog's `--log-file` parameter was supposed to fix this problem, but we ran into trouble with that in scalability CI jobs that never could get root caused and fixed. Using this binary worked. - Windows services don't have a mechanism for redirecting output of a process. - Nowadays, the `--log-file` parameter is deprecated for Kubernetes components and should not be used anymore. `kube-log-runner` is a direct replacement. For example instead of running kube-apiserver like this: ```bash "/bin/sh", "-c", "exec kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" ``` Or this: ```bash kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} --log-file=/var/log/kube-apiserver.log --alsologtostderr=false" ``` We would use `kube-log-runner` like so: ```bash kube-log-runner -log-file=/var/log/kube-apiserver.log --also-stdout=false  kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} ``` The kube-log-runner then ensures that we run the `/usr/local/bin/kube-apiserver` with the specified parameters and redirect both stdout and stderr ONLY to the log file specified. It will always append to the log file. Possible invocations: ```bash # Merge stderr and stdout, write to stdout (same as 2>&1). kube-log-runner echo "hello world" # Redirect both into log file (same as 1>>/tmp/log 2>&1). kube-log-runner -log-file=/tmp/log echo "hello world" # Copy into log file and print to stdout (same as 2>&1 | tee -a /tmp/log). kube-log-runner -log-file=/tmp/log -also-stdout echo "hello world" # Redirect only stdout into log file (same as 1>>/tmp/log). kube-log-runner -log-file=/tmp/log -redirect-stderr=false echo "hello world" ``` # Container base image The Kubernetes [`k8s.gcr.io/build-image/go-runner`](https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/us/build-image/go-runner) image wraps the `gcr.io/distroless/static` image and provides `kube-log-runner` under its traditional name as `/go-runner`. It gets maintained in https://github.com/kubernetes/release/tree/master/images/build/go-runner. # Prebuilt binary The Kubernetes release archives contain kube-log-runner. </ins>

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for windows [ ] [ ] /sig windows /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I've started looking into this. Part of the problem is that go-runner is not part of kubernetes-node-windows- I suspect that is where the scripts get and kube- from.


Question: requests := make(map[string]int64, len(pods)) for _, pod := range pods { podSum := int64(0) <del> for _, c := range pod.Spec.Containers { </del> <ins> // Calculate all regular containers and restartable init containers requests. containers := append([]v1.Container{}, pod.Spec.Containers...) for _, c := range pod.Spec.InitContainers { if c.RestartPolicy != nil && *c.RestartPolicy == v1.ContainerRestartPolicyAlways { containers = append(containers, c) } } for _, c := range containers { </ins> if container == "" || container == c.Name { if containerRequest, ok := c.Resources.Requests[resource]; ok { podSum += containerRequest.MilliValue()

Calculate sidecar containers resource in pod autoscaler. From 1.28 user could run Sidecar containers in , those sidecar containers remain running during the entire life of the Pod, it should be calculated in . If missing the sidecar resources, we will get a wrong pod resource utilization. We should also correct the related Cadvisor metrics.
/assign /sig autoscaling
/triage accepted /priority important-soon


Question: }, field.ErrorList{{Type: field.ErrorTypeInvalid, Field: "containers[0].envFrom[0].configMapRef.name"}}, }, <ins> { "Unsupported resize policy for memory", line(), []core.Container{ { Name:                     "resize-policy-mem-invalid", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", ResizePolicy: []core.ContainerResizePolicy{ {ResourceName: "memory", RestartPolicy: "RestartContainerrrr"}, }, }, }, field.ErrorList{{Type: field.ErrorTypeNotSupported, Field: "containers[0].resizePolicy"}}, }, { "Unsupported resize policy for CPU", line(), []core.Container{ { Name:                     "resize-policy-cpu-invalid", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", ResizePolicy: []core.ContainerResizePolicy{ {ResourceName: "cpu", RestartPolicy: "RestartNotRequired"}, }, }, }, field.ErrorList{{Type: field.ErrorTypeNotSupported, Field: "containers[0].resizePolicy"}}, }, </ins> } for _, tc := range errorCases { t.Run(tc.title+"__@L"+tc.line, func(t *testing.T) {

The newly field field is not being validated. field should be validated. We can create a pod with invalid to reproduce this. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /sig node /priority important-soon /milestone v1.27 adding to 1.27 milestone to resolve if possible
I'll try this for v1.27 if you don't mind.
getting a PR open, reviewed, and green is always good. we'll evaluate it for risk and if there's a second release candidate we can likely include it
Nice catch! Please review fix: This likely got dropped either during port-over of the or during bad rebase. I should not have left the . Gotta run to get my kid prepped, dr. appt and then school drop. So I'll try and add an E2E test for this later today. please hold for e2e test (but we should fix this in 1.27)
I realized you were working on a fix as well only after I sent my fix out and was catching up on emails at doc office (rushed morning). I didn't mean to step over your effort to fix it, sorry about that.
I am completely fine with that. Thanks for the fix :)


Question: expc.recorder.Event(pvc, v1.EventTypeNormal, events.ExternalExpanding, msg) csiResizerName, err := expc.translator.GetCSINameFromInTreeName(inTreePluginName) if err != nil { <del> errorMsg := fmt.Sprintf("error getting CSI driver name for pvc %s, with error %v", util.ClaimToClaimKey(pvc), err) </del> <ins> errorMsg := fmt.Sprintf("error getting CSI driver name for pvc %s, with error %v", key, err) </ins> expc.recorder.Event(pvc, v1.EventTypeWarning, events.ExternalExpanding, errorMsg) return fmt.Errorf(errorMsg) } pvc, err := util.SetClaimResizer(pvc, csiResizerName, expc.kubeClient) if err != nil { <del> errorMsg := fmt.Sprintf("error setting resizer annotation to pvc %s, with error %v", util.ClaimToClaimKey(pvc), err) </del> <ins> errorMsg := fmt.Sprintf("error setting resizer annotation to pvc %s, with error %v", key, err) </ins> expc.recorder.Event(pvc, v1.EventTypeWarning, events.ExternalExpanding, errorMsg) return fmt.Errorf(errorMsg) }

I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.


Question: } if kl.nodeIP != nil { enforcedNodeAddresses := []v1.NodeAddress{} <ins> var nodeIPType v1.NodeAddressType </ins> for _, nodeAddress := range nodeAddresses { if nodeAddress.Address == kl.nodeIP.String() { enforcedNodeAddresses = append(enforcedNodeAddresses, v1.NodeAddress{Type: nodeAddress.Type, Address: nodeAddress.Address}) <ins> nodeIPType = nodeAddress.Type break </ins> } } if len(enforcedNodeAddresses) > 0 { <ins> for _, nodeAddress := range nodeAddresses { if nodeAddress.Type != nodeIPType && nodeAddress.Type != v1.NodeHostName { enforcedNodeAddresses = append(enforcedNodeAddresses, v1.NodeAddress{Type: nodeAddress.Type, Address: nodeAddress.Address}) } } </ins> enforcedNodeAddresses = append(enforcedNodeAddresses, v1.NodeAddress{Type: v1.NodeHostName, Address: kl.GetHostname()}) node.Status.Addresses = enforcedNodeAddresses return nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: <ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package benchmark import ( "fmt" "testing" "time" ) // TestSchedule100Node3KPods schedules 3k pods on 100 nodes. func TestSchedule100Node3KPods(t *testing.T) { schedulePods(100, 3000) } // TestSchedule1000Node30KPods schedules 30k pods on 1000 nodes. func TestSchedule1000Node30KPods(t *testing.T) { schedulePods(1000, 30000) } // schedulePods schedules specific number of pods on specific number of nodes. // This is used to learn the scheduling throughput on various // sizes of cluster and changes as more and more pods are scheduled. // It won't stop until all pods are scheduled. func schedulePods(numNodes, numPods int) { schedulerConfigFactory, destroyFunc := mustSetupScheduler() defer destroyFunc() c := schedulerConfigFactory.Client makeNodes(c, numNodes) makePods(c, numPods) prev := 0 start := time.Now() for { // This can potentially affect performance of scheduler, since List() is done under mutex. // Listing 10000 pods is an expensive operation, so running it frequently may impact scheduler. // TODO: Setup watch on apiserver and wait until all pods scheduled. scheduled := schedulerConfigFactory.ScheduledPodLister.Store.List() fmt.Printf("%dstrate: %dttotal: %dn", time.Since(start)/time.Second, len(scheduled)-prev, len(scheduled)) if len(scheduled) >= numPods { return } prev = len(scheduled) time.Sleep(1 * time.Second) } } </ins>

We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.

+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


Question: package main import ( <del> "errors" </del> goflag "flag" <del> "fmt" </del> "math/rand" "os" <del> "path" </del> "path/filepath" "time"

Follow up for the deprecation we did in 1.14:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
I am willing to take this, if it is available :)
go for it. All you need to do is remove and associated code -
I ended up getting hit with the removal of the --make-symlinks flag. I looked at the threads linked to the original PR, but I did not find the actual reason behind the removal. Do you recall was the reason behind removing that capability?
the original request was from


Question: if err != nil { return err } <del> mustCheckData = false continue </del> <ins> if !bytes.Equal(data, origState.data) { // original data changed, restart loop mustCheckData = false continue } </ins> } return decode(s.codec, s.versioner, origState.data, out, origState.rev) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRD: Existing object: New object: says: If this is a problem with the new object definition rather than a bug with , I'd expect a better error message. (The new object looks fine to me, though; it should be identical to the old object.) What you expected to happen: I expected the object to be updated without an error. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig api-machinery
Are you able to reproduce this everytime? I am able to use apply normally with your example.
I can reliably reproduce it if the annotation exists: If I use to remove the annotation, then it works the first time but fails thereafter:
This is happening for other CRDs as well (Prometheus Operator ServiceMonitors). If I apply a ServiceMonitor CRD 3 times: If I remove the annotation or replace the object () then I can work around the issue.
Seems any CRD is affected; the CRD used by our are also not working: This has completely broken our CI/CD workflow... unfortunately we'd already completed the TPR-CRD migration for the 1.8 upgrade before noticing this problem, so we can't really downgrade. How safe is as an alternative to ? It still generates an update event for watchers, not a delete/recreate, right?
It looks like something going wrong during the patching. After using changes from , I get this: /cc /area third-party-resource
But uses json merge patch and not strategic merge patch for custom resources:
I was able to reproduce this in v1.8.0 with the original manifests at the top. As found, it only happens on the 3rd execution: It also seems to only happen when the ought to be a no-op.

If I change something in the IPPool spec, works once more, then starts failing again. When the is a no-op, the PATCH request body generated by looks like this: When the is NOT a no-op, the PATCH request body looks like this:
There's a typo in the code generating the error messages for both and : They should be calling instead of . When I fix that, the error message becomes:
I can reproduce this without , just by doing a no-op twice:
The same issue affects v1.7.8 (tested on GKE)
Same issue with v1.8.1
This just fix the error message: But it doesn't fix the issue
Still investigating the issue but I noticed that when we do a no-op patch for native resources, we get a "not patched" response. However, for non-native resources, it still tries to patch it (and bumps the resourceVersion too).
I don't konwn whether this is a issue, I can't do strategic merge patch to CRD. When I run , --type=merge must be passed.
We should be able to apply strategic merge patch to CRDs. However, we can't apply strategic merge to CRs because we're missing the metadata about how to handle each field for the strategic merge patch. Unfortunately, this error is occuring even with . :(
/assign
do you know if the fix fir this will make it to 1.8.3
hopefully yes, this should be merged soon.
thanks! this has been a blocker for us to upgrade to 1.8
can you confirm this is completely fixed? The PR writes "partial fix".
Now we can't reproduce this issue with the method descripted above. I still need some I extra time to research it. Maybe we can close this one. If I can reproduce the issue in another way, I can open another issue with new situation.
If it's not fixed, let's keep it open.
Fun, a fork closing issues.
From Then I think we can officially close this.
(the fix is merged into release-1.8, but after v1.8.4-beta.0)
what stable release of k8s will this be in? 1.8.4 or 1.8.5?
Is there an update for this?

looks like it was merged here: but did not make it into 1.8.4, might make it into 1.8.5


Question: // NOTE: events received on your eventHandler should be copied before being used. // TODO: figure out if this can be removed. StartEventWatcher(eventHandler func(event runtime.Object)) func() <ins> // Shutdown shuts down the broadcaster Shutdown() </ins> } // EventSink knows how to store events (client-go implements it.)

In order to initialize a client-go , you need an . and friends initialize a new : launches a goroutine: This goroutine is stoppable with . However, the interface returned above does not expose anything that calls that shutdown method, AFAICT. This means that that goroutine cannot ever be cleaned up. That's not great. It's fine if you're using in like k/k does and never stopping anything, but if you ever want to stop your controllers and start up new ones, it becomes an issue. Tested/tracked down against master (as of ). /sig apimachinery /kind bug
The label(s) cannot be applied. These labels are supported: <detailsIn response to : In order to initialize a client-go , you need an . and friends initialize a new : /kind bug </details
/sig api-machinery I always forget the dash


Question: } } else { // If volume is not safe to detach (is mounted) wait a max amount of time before detaching any way. <del> timeElapsed, err := rc.actualStateOfWorld.MarkDesireToDetach(attachedVolume.VolumeName, attachedVolume.NodeName) if err != nil { glog.Errorf("Unexpected error actualStateOfWorld.MarkDesireToDetach(): %v", err) } </del> if timeElapsed > rc.maxWaitForUnmountDuration { glog.V(5).Infof("Attempting to start DetachVolume for volume %q from node %q. Volume is not safe to detach, but maxWaitForUnmountDuration expired.", attachedVolume.VolumeName, attachedVolume.NodeName) err := rc.attacherDetacher.DetachVolume(attachedVolume.AttachedVolume, rc.actualStateOfWorld)

introduced logic to make kubelet volume manager wait for attach detach controller to finish attach before it starts mount. The controller should also not start a detach operation until the node status has been updated removing the volume from the list of attached volumes. Otherwise if a pod is rapidly created and deleted on the same node, it could result in a mount starting immediately before a detach starts. Props to and his pet set tests for helping catch this.
Hmm, this just happened , just a lot less frequently. Haven't debugged, but it looks like the failed pod did reland on the same node: pet-0, jenkins-e2e-minion-group-4mn2 (). Reopening, will try to confirm when i have time, and close if it's not the same thing.
Just following up with an update, Saad and i are tag teaming this. He's uploading a pr that I'm going to reivew. We're tracking a couple of issues: need to update the "volumes in use" list before polling on the "volumes attached" list. Best option is to write a operation executor, so we don't hammer the apiserver. needs to get up to date node object from apiserver to determine volumes in use. The update that removes volumes from the "attached list" shoud return a fresh node object, we can just iterate the volumes its using (this must happen after popping from attached, though). leaky volumes in the node object (this is not about the physical volumes but about the size of the json blob that's the node object growing). The proposed fix is to use the desired state of the world to update the list of volumes in use. There's a last point I need to follow up on. We might leave the node object in an inconsistent state with the proposed fix, but it should be ok since nothing is checking the inconsistent fields. This will happen if we pop a volume from the attached list, but then find its in use in the updated node object. ping me when you're ready to go, of if you need help


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <del> "k8s.io/apimachinery/pkg/runtime" </del> <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/runtime/serializer" "k8s.io/apiserver/pkg/endpoints/handlers/responsewriters" v1listers "k8s.io/client-go/listers/core/v1"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): build--5-v1.7.5-3 Environment: Cloud provider or hardware configuration: Laptop (bare metal) OS (e.g. from /etc/os-release): Fedora 26 Kernel (e.g. ): Linux localhost.localdomain 4.11.0-0.rc6.git0.1.fc26.x8664 SMP Mon Apr 10 19:45:50 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: make && hack/local-cluster- -o _output/bin What happened: built from and ran with local-cluster- All commands gave the following error: Running showed the output: What you expected to happen: for commands to work, and Content-type to show How to reproduce it (as minimally and precisely as possible):
ccing
I see the text plain content type, but not kubectl errors
i was using the wrong version of kubectl. updated to new and i'm not getting an error any longer.
Text/plain coming back from the server is still an issue
cc
I'm not seeing against head and the pastebin is gone. Did you reproduce?
I did last week. Insecure and secured port behaved differently
You can reproduce this using against a Kubernetes cluster built from HEAD
Hi all, I think we could use write Content-type correctly. I have made a PR to fix this.