Question: scheduler.WithPodMaxBackoffSeconds(0), ) testutils.SyncInformerFactory(testCtx) <del> // It's intended to not start the scheduler's queue, and hence to // not start any flushing logic. We will pop and schedule the Pods manually later. </del> defer testutils.CleanupTest(t, testCtx) cs, ns, ctx := testCtx.ClientSet, testCtx.NS.Name, testCtx.Ctx

Currently, uses ServiceAffinity plugin to cover the event registration logic: But along with the removal of ServiceAffinity plugin, this test will also be removed . We will come up with a supplemental test to cover the event registration logic. /sig scheduling
/assign /triage accepted
/kind cleanup


Question: glog.Warningf("Flag proxy-mode=%q unknown, assuming iptables proxy", proxyMode) return tryIPTablesProxy(iptver, kcompat) } <del> // proxyMode == "" - choose the best option. if client == nil { glog.Errorf("nodeGetter is nil: assuming iptables proxy") return tryIPTablesProxy(iptver, kcompat) } node, err := client.Get(hostname, metav1.GetOptions{}) if err != nil { glog.Errorf("Can't get Node %q, assuming iptables proxy, err: %v", hostname, err) return tryIPTablesProxy(iptver, kcompat) } if node == nil { glog.Errorf("Got nil Node %q, assuming iptables proxy", hostname) return tryIPTablesProxy(iptver, kcompat) } proxyMode, found := node.Annotations[betaProxyModeAnnotation] if found { glog.V(1).Infof("Found beta annotation %q = %q", betaProxyModeAnnotation, proxyMode) } else { // We already published some information about this annotation with the "experimental" name, so we will respect it. proxyMode, found = node.Annotations[experimentalProxyModeAnnotation] if found { glog.V(1).Infof("Found experimental annotation %q = %q", experimentalProxyModeAnnotation, proxyMode) } } if proxyMode == proxyModeUserspace { glog.V(1).Infof("Annotation demands userspace proxy") return proxyModeUserspace } </del> return tryIPTablesProxy(iptver, kcompat) }

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: curl -fs ${SWAGGER_API_PATH}version > ${SWAGGER_ROOT_DIR}/version.json curl -fs ${SWAGGER_API_PATH}api > ${SWAGGER_ROOT_DIR}/api.json curl -fs ${SWAGGER_API_PATH}api/v1 > ${SWAGGER_ROOT_DIR}/v1.json <ins> curl -fs ${SWAGGER_API_PATH}apis > ${SWAGGER_ROOT_DIR}/apis.json curl -fs ${SWAGGER_API_PATH}apis/extensions > ${SWAGGER_ROOT_DIR}/extensions.json </ins> curl -fs ${SWAGGER_API_PATH}apis/extensions/v1beta1 > ${SWAGGER_ROOT_DIR}/v1beta1.json kube::log::status "SUCCESS"

Loading returns Looking at the code is registered in the same way as which works fine. So not sure why is this broken.


Question: close(finished) return devicePath, nil } <ins> fakePlugin.Unlock() </ins> // Start the reconciler again. go reconciler.Run(wait.NeverStop)

Which jobs are flaking: Which test(s) are flaking: periodic-kubernetes-bazel-test-master (xref ) Testgrid link: Reason for failure: Anything else we need to know: to reproduce with : /sig storage
/assign
solved at
still flaking - /reopen
Reopened this issue. <detailsIn response to : still flaking - Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
-- No recent failures in the past 2 weeks, are you okay if we close this?
yeah, looks like resolved it /close
Closing this issue. <detailsIn response to : yeah, looks like resolved it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's back not sure if related to this issue but same test with a data race spotted last night looking into filed


Question: config, err := options.loadConfig([]byte(yaml)) assert.NoError(t, err, "unexpected error for %s: %v", tc.name, err) if !reflect.DeepEqual(expected, config) { <del> t.Fatalf("unexpected config for %s test, diff = %s", tc.name, diff.ObjectDiff(config, expected)) </del> <ins> t.Fatalf("unexpected config for %s, diff = %s", tc.name, diff.ObjectDiff(config, expected)) </ins> } } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: func compareCacheWithNodeInfoSnapshot(t *testing.T, cache *schedulerCache, snapshot *Snapshot) error { // Compare the map. <del> if len(snapshot.nodeInfoMap) != len(cache.nodes) { return fmt.Errorf("unexpected number of nodes in the snapshot. Expected: %v, got: %v", len(cache.nodes), len(snapshot.nodeInfoMap)) </del> <ins> if len(snapshot.nodeInfoMap) != cache.nodeTree.numNodes { return fmt.Errorf("unexpected number of nodes in the snapshot. Expected: %v, got: %v", cache.nodeTree.numNodes, len(snapshot.nodeInfoMap)) </ins> } for name, ni := range cache.nodes { <del> if !reflect.DeepEqual(snapshot.nodeInfoMap[name], ni.info) { return fmt.Errorf("unexpected node info for node %q. Expected: %v, got: %v", name, ni.info, snapshot.nodeInfoMap[name]) </del> <ins> want := ni.info if want.Node() == nil { want = nil } if !reflect.DeepEqual(snapshot.nodeInfoMap[name], want) { return fmt.Errorf("unexpected node info for node %q.Expected:n%v, got:n%v", name, ni.info, snapshot.nodeInfoMap[name]) </ins> } } // Compare the lists. <del> if len(snapshot.nodeInfoList) != len(cache.nodes) { return fmt.Errorf("unexpected number of nodes in NodeInfoList. Expected: %v, got: %v", len(cache.nodes), len(snapshot.nodeInfoList)) </del> <ins> if len(snapshot.nodeInfoList) != cache.nodeTree.numNodes { return fmt.Errorf("unexpected number of nodes in NodeInfoList. Expected: %v, got: %v", cache.nodeTree.numNodes, len(snapshot.nodeInfoList)) </ins> } expectedNodeInfoList := make([]*framework.NodeInfo, 0, cache.nodeTree.numNodes)

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: return desiredVolumes } <del> func (kl *Kubelet) cleanupOrphanedPodDirs(pods []*api.Pod) error { desired := util.NewStringSet() </del> <ins> // cleanupOrphanedPodDirs removes a pod directory if the pod is not in the // desired set of pods and there is no running containers in the pod. func (kl *Kubelet) cleanupOrphanedPodDirs(pods []*api.Pod, runningPods []*kubecontainer.Pod) error { active := util.NewStringSet() </ins> for _, pod := range pods { <del> desired.Insert(string(pod.UID)) </del> <ins> active.Insert(string(pod.UID)) </ins> } <ins> for _, pod := range runningPods { active.Insert(string(pod.ID)) } </ins> found, err := kl.listPodsFromDisk() if err != nil { return err } errlist := []error{} for i := range found { <del> if !desired.Has(string(found[i])) { </del> <ins> if !active.Has(string(found[i])) { </ins> glog.V(3).Infof("Orphaned pod %q found, removing", found[i]) if err := os.RemoveAll(kl.getPodDir(found[i])); err != nil { errlist = append(errlist, err)

Often observed in logs: However: Which probably means we'd be ok unmounting it: This probably also means we don't delete orphaned mirror pods, which is the next step in SyncPods ().


Question: - endpoint: createAuthenticationV1SelfSubjectReview reason: Cluster providers are allowed to choose to not serve this API, and the whoami command handles unavailability gracefully. link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/3325-self-subject-attributes-review-api/README.md#ga <ins> - endpoint: getInternalApiserverAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 - endpoint: getResourceAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 - endpoint: getStoragemigrationAPIGroup reason: The endpoint is part of a group that has no stable endpoints yet, reporting as a stable endpoint in apisnoop. link: https://github.com/kubernetes/issues/124248 </ins>

The following api endpoints are currently in - As each group has no stable endpoints they are reported in as promoted to stable without tests. The current logic for endpoints will address this issue.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture /area test


Question: q.ShutDown() consumerWG.Wait() } <ins> func TestLen(t *testing.T) { q := workqueue.New() q.Add("foo") if e, a := 1, q.Len(); e != a { t.Errorf("Expected %v, got %v", e, a) } q.Add("bar") if e, a := 2, q.Len(); e != a { t.Errorf("Expected %v, got %v", e, a) } q.Add("foo") // should not increase the queue length. if e, a := 2, q.Len(); e != a { t.Errorf("Expected %v, got %v", e, a) } } </ins>

So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.


Question: return allErrs } <ins> // AccumulateUniqueHostPorts checks all the containers for duplicates ports. Any // duplicate port will be returned in the ErrorList. </ins> func AccumulateUniqueHostPorts(containers []v1.Container, accumulator *sets.String, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{}

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } } <ins> func TestTruncatePodHostname(t *testing.T) { for c, test := range map[string]struct { input  string output string }{ "valid hostname": { input:  "test.pod.hostname", output: "test.pod.hostname", }, "too long hostname": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567.", // 8*9=72 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.1234567",          //8*8-1=63 chars }, "hostname end with .": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456.1234567.", // 8*9-1=71 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456",          //8*8-2=62 chars }, "hostname end with -": { input:  "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456-1234567.", // 8*9-1=71 chars output: "1234567.1234567.1234567.1234567.1234567.1234567.1234567.123456",          //8*8-2=62 chars }, } { t.Logf("TestCase: %q", c) output, err := truncatePodHostnameIfNeeded("test-pod", test.input) assert.NoError(t, err) assert.Equal(t, test.output, output) } } </ins>

Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here


Question: // waitForAttachmentStatus polls until the attachment status is the expected value // On success, it returns the last attachment state. func (d *awsDisk) waitForAttachmentStatus(status string) (*ec2.VolumeAttachment, error) { <del> // We wait up to 30 minutes for the attachment to complete. // This mirrors the GCE timeout. timeoutAt := time.Now().UTC().Add(volumeAttachmentStatusTimeout).Unix() </del> <ins> backoff := wait.Backoff{ Duration: volumeAttachmentStatusInitialDelay, Factor:   volumeAttachmentStatusFactor, Steps:    volumeAttachmentStatusSteps, } </ins> // Because of rate limiting, we often see errors from describeVolume // So we tolerate a limited number of failures. // But once we see more than 10 errors in a row, we return the error describeErrorCount := 0 <ins> var attachment *ec2.VolumeAttachment </ins> <del> for { </del> <ins> err := wait.ExponentialBackoff(backoff, func() (bool, error) { </ins> info, err := d.describeVolume() if err != nil { describeErrorCount++ if describeErrorCount > volumeAttachmentStatusConsecutiveErrorLimit { <del> return nil, err </del> <ins> // report the error return false, err </ins> } else { glog.Warningf("Ignoring error from describe volume; will retry: %q", err) <del> time.Sleep(volumeAttachmentStatusErrorDelay) continue </del> <ins> return false, nil </ins> } } else { describeErrorCount = 0

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.


Question: }, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 30 scheduler_pending_pods{queue="backoff"} 0

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: {% if pillar.get('network_provider', '').lower() == 'opencontrail' %} {% set network_plugin = "--network-plugin=opencontrail" %} {% elif pillar.get('network_provider', '').lower() == 'cni' %} <del> {% set network_plugin = "--network-plugin=cni --network-plugin-dir=/etc/cni/net.d/" %} </del> <ins> {% set network_plugin = "--network-plugin=cni --cni-bin-dir=/etc/cni/net.d/" %} </ins> {%elif pillar.get('network_policy_provider', '').lower() == 'calico' and grains['roles'][0] != 'kubernetes-master' -%} <del> {% set network_plugin = "--network-plugin=cni --network-plugin-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} </del> <ins> {% set network_plugin = "--network-plugin=cni --cni-conf-dir=/etc/cni/net.d/ --cni-bin-dir=/home/kubernetes/bin/" %} </ins> {% elif pillar.get('network_provider', '').lower() == 'kubenet' %} {% set network_plugin = "--network-plugin=kubenet" -%} {% endif -%}

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: "path/filepath" "regexp" "strings" <ins> "sync" </ins> "time" "github.com/spf13/pflag"

/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!


Question: "io" "math/rand" "reflect" <ins> "strings" </ins> "sync" "time"

What happened: Using client-go v1.18.8 to watch resources on API server versions 1.17.x and 1.18.0 - 1.18.5 After the connection to the server is temporarily interrupted the client never recovers from error See the earlier issue for steps to reproduce: This was assumed fixed by But it appears the fix for the client was (probably inadvertently) implemented in a non-backward-compatible fashion and only solves the issue when the server is running v1.18.6 or later as well.
/sig api-machinery
Cc
PR
We generally recommend having apiservers be updated first before all other components. But yeah - it's true that this doesn't work for clients being more fresh - let me take a look at your PR.
/cc
This will be addressed by (and the cherrypicking it back).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed by and corresponding cherrypicks.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine.

This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?

got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones.

But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.

Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?


Question: import "fmt" <del> func GetVMUUID() (string, error) { </del> <ins> func getRawUUID() (string, error) { </ins> return "", fmt.Errorf("Retrieving VM UUID on this build is not implemented.") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running the vSphere cloud provider on a windows node throws the following error: This is because the vm identifier is retrieved with the command - , which assumes that the uuid and serial are the same. On some vmware vm versions, the uuid and serial are not the same. vmware (comment), vmware (comment) What you expected to happen: Kubelet is able to find vm in vCenter from the identifier looked up on the node. or should be used instead to find the vm identifier. How to reproduce it (as minimally and precisely as possible): Issue only occurs on some vm versions, such as 13. Anything else we need to know?: On the linux side, this issue was resolved by switching from looking up the id from to . See Some discussion about this issue here - Environment: Kubernetes version (use ): 14.0-beta.1 Cloud provider or hardware configuration: Vsphere OS (e.g: ): Windows 2019 Kernel (e.g. ): Install tools: Others:
/sig windows /sig vmware
/milestone v1.14
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.14 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: // in the test anyway. continue } <del> // If the image policy is not PullAlways, the image must be in the white list and </del> <ins> // If the image policy is not PullAlways, the image must be in the pre-pull list and </ins> // pre-pulled. <del> gomega.Expect(ImageWhiteList.Has(c.Image)).To(gomega.BeTrue(), "Image %q is not in the white list, consider adding it to CommonImageWhiteList in test/e2e/common/util.go or NodeImageWhiteList in test/e2e_node/image_list.go", c.Image) // Do not pull images during the tests because the images in white list should have </del> <ins> gomega.Expect(ImageWhiteList.Has(c.Image)).To(gomega.BeTrue(), "Image %q is not in the white list, consider adding it to CommonImageWhiteList in test/e2e/common/util.go or NodePrePullImageList in test/e2e_node/image_list.go", c.Image) // Do not pull images during the tests because the images in pre-pull list should have </ins> // been prepulled. c.ImagePullPolicy = v1.PullNever }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should come up with a better name for : Why is this needed: From /sig testing /priority important-soon /cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --We should come up with a better name for : From /priority important-soon /cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to help out with this. /assign
mind. It seems like you have a PR for it already. /unassign
And I misread the changes you made in . I'm back on this one. /assign
I'm going to rename this to . Please let me know if there are any objections or if there are things that I should be considering.
please feel free to take it. I'm only changing ImageWhileList withint
ultimately was changed to NodePrePullImageList because that's what it's actually used for, not for any sort of filtering.


Question: socket string stop   chan interface{} <ins> wg     sync.WaitGroup </ins> update chan []*pluginapi.Device server *grpc.Server

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: defer rc.CleanUp() hpa := e2eautoscaling.CreateCPUHorizontalPodAutoscalerWithBehavior( <del> rc, int32(targetCPUUtilizationPercent), 1, 5, e2eautoscaling.HPABehaviorWithDownscaleStabilizationSeconds(int32(downScaleStabilization.Seconds())), </del> <ins> rc, int32(targetCPUUtilizationPercent), 1, 5, int32(downScaleStabilization.Seconds()), </ins> ) defer e2eautoscaling.DeleteHPAWithBehavior(rc, hpa.Name)

gce-cos-autoscaling-hpa-cpu Kubernetes e2e suite.[sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with long upscale stabilization window should scale up only after the stabilization period Since it was in (first run at 2022-06-14 13:13 CEST failed, all subsequent runs failed too). https://k8s- Times out when waiting for scale down (), logs from (): I'm not sure why exactly, I have 2 ideas: It looks like the consumer briefly consumes 440 mCPU, which could result in a higher recommendation being generated HPA stabilises initial size of the consumer Increasing timeout should fix the issue No response_ /sig autoscaling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI
This was already reverted (and later the test got fixed and merged). /close
Closing this issue. <detailsIn response to : This was already reverted (and later the test got fixed and merged). Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } wg.Add(1) go func(index int) { <del> defer wg.Done() </del> <ins> metrics.SchedulerGoroutines.WithLabelValues("prioritizing_mapreduce").Inc() defer func() { metrics.SchedulerGoroutines.WithLabelValues("prioritizing_mapreduce").Dec() wg.Done() }() </ins> if err := priorityConfigs[index].Reduce(pod, meta, nodeNameToInfo, results[index]); err != nil { appendError(err) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


Question: "github.com/onsi/ginkgo" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1" <del> rbacv1 "k8s.io/api/rbac/v1" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels"

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: if [[ ${#block_devices[@]} == 0 ]]; then echo "No ephemeral block devices found; will use aufs on root" docker_storage="aufs" <del> # Install aufs kernel module (for ubuntu) apt-get install --yes linux-image-extra-$(uname -r) # Install aufs tools (for debian) apt-get install --yes aufs-tools </del> else echo "Block devices: ${block_devices[@]}"

On AWS with Vivid, auplink is not installed. There are messages in the docker log about auplink not being in the path. Inspection of the format-disks script confirms this to be the case. (PR coming soon, just opening an issue so we have something to track it back to!)


Question: kubectl get rc,services # List one or more resources by their type and names. <del> kubectl get rc/web service/frontend pods/web-pod-13je7 # List all resources with different types. kubectl get all`)) </del> <ins> kubectl get rc/web service/frontend pods/web-pod-13je7`)) </ins> ) const (

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig cli What happened: I have two custom resources in my cluster called Cluster, and couldn't figure out how to ask for one or the other. was of no help, and various Google searches for things like "kubectl apigroup" or "kubectl crd" revealed nothing of use. Worse, some bugs that came up like suggested that the syntax was , which produced entirely incorrect output (e.g. results in . Eventually, I stumbled by accident on the correct way to do it, which is to do and then , where is the name of the custom resource exactly as listed in the output of . I still don't know how to request a specific API version using this syntax, if that is even possible. What you expected to happen: I expected to document how to query custom resources, given that they're increasingly central to running production quality clusters with various operators. Failing that, I expected the k8s website to have documentation on querying custom resources that is discoverable via Google search. How to reproduce it (as minimally and precisely as possible): Install something that uses custom resources (e.g. Calico in k8s datastore mode, Rook), and try to one of the custom resources by precise name (e.g. assume that there are multiple objects defined by different CRDs, so you can't just , you have to be more precise). Environment: Kubernetes version (use ): 1.10.2 client+cserver
Opened with examples


Question: func (tcp *tcpProxySocket) ProxyLoop(service proxy.ServicePortName, myInfo *serviceInfo, proxier *Proxier) { for { <del> if info, exists := proxier.getServiceInfo(service); !exists || info != myInfo { </del> <ins> if !myInfo.isAlive() { </ins> // The service port was closed or replaced. return }

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: Command: []string{ "sh", "-c", <del> "trap "echo SIGTERM caught" SIGTERM SIGINT; touch /tmp/healthy; sleep 1000", </del> <ins> "trap "echo SIGTERM caught" SIGTERM SIGINT; touch /tmp/healthy; /bin/sleep 1000", </ins> }, // Using readiness probe to guarantee signal handler registering finished ReadinessProbe: &v1.Probe{

After is fixed, there are still several failing cases. This one is one of them. /kind failing-test /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Super weird, still looking but appears to be segfaulting. but it works fine if I manually run the container with , I've tried both with and without a TTY.
Switching from to eliminated the segfault and the test passes. The binary from the 1.29-4 container works fine on this system outside of the pod, and works fine using a container manually started with . Trying to identify why it fails with 1.29-4 when run from a pod, but not ctr directly. I built a busybox top v1.29 binary with debugging information into a new image to debug further, and no crash.
HUH!!! wow.


Question: detect-master <del> ./target/cloudcfg -h https://${KUBE_MASTER_IP} $@ </del> <ins> $(dirname $0)/../../target/cloudcfg -h https://${KUBE_MASTER_IP} $@ </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The image () used to run the per-pod infra/pause container logs to stdout about once per second. What you expected to happen: Pause container not logging anything. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Kubelet does not expose, nor manage these logs. Writing the logs wastes resources and takes up disk space, which may not be properly accounted by kubelet. The pause image used for the e2e tests seems much better: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign Will also rebuild with


Question: link += "http://" + ip + ":" + strconv.Itoa(int(port.Port)) + " " } } else { <ins> name := service.ObjectMeta.Name if len(service.Spec.Ports) > 0 { port := service.Spec.Ports[0] // guess if the scheme is https scheme := "" if port.Name == "https" || port.Port == 443 { scheme = "https" } // format is <scheme>:<service-name>:<service-port-name> name = utilnet.JoinSchemeNamePort(scheme, service.ObjectMeta.Name, port.Name) } </ins> if len(client.GroupVersion.Group) == 0 { <del> link = client.Host + "/api/" + client.GroupVersion.Version + "/namespaces/" + service.ObjectMeta.Namespace + "/services/" + service.ObjectMeta.Name + "/proxy" </del> <ins> link = client.Host + "/api/" + client.GroupVersion.Version + "/namespaces/" + service.ObjectMeta.Namespace + "/services/" + name + "/proxy" </ins> } else { <del> link = client.Host + "/api/" + client.GroupVersion.Group + "/" + client.GroupVersion.Version + "/namespaces/" + service.ObjectMeta.Namespace + "/services/" + service.ObjectMeta.Name + "/proxy" </del> <ins> link = client.Host + "/api/" + client.GroupVersion.Group + "/" + client.GroupVersion.Version + "/namespaces/" + service.ObjectMeta.Namespace + "/services/" + name + "/proxy" </ins> } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubectl cluster-info displays proxy urls for cluster services If those services are using https, the referenced urls do not work (they try to proxy using http) What you expected to happen: Services with a port name of https or 443 should use the ‘https:<name:<port‘ form How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /sig cli


Question: }, expected: false, }, <ins> { cfg: &kubeadmapi.MasterConfiguration{ API: kubeadmapi.API{AdvertiseAddress: "2001:1234::1:15"}, }, expected: false, }, </ins> } for _, rt := range tests {

BUG REPORT /kind bug What happened: Kubeadm checks that bridge-nf-call-iptables (IPv4) is set, but does not check for bridge-nf-call-ip6tables (IPv6) What you expected to happen: If IPv6 is being used, then Kubeadm should check for bridge-nf-call-ip6tables How to reproduce it (as minimally and precisely as possible): Just running Kubeadm, however it's only a problem when running IPv6 Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area ipv6 /area kubeadm
/sig network


Question: detect-master parse-master-env <ins> backfile-kubeletauth-certs </ins> # Delete the master instance. Note that the master-pd is created # with auto-delete=no, so it should not be deleted.

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug What happened: Upgrade hangs. API server won't start because it doesn't have valid certs. What you expected to happen: Upgrade succeeds How to reproduce it (as minimally and precisely as possible): Anything else we need to know: https://k8s- is an example. Log from kube- I reproduced this locally and checked the certs: apiserver.{crt,key} are empty.
cc
Looks like this is still failing? https://k8s-
Seems that way.
cc
5, to 1.6
- is this something you are actively looking into? This is blocking all of the GCE upgrade test jobs from running so important to get fixed ASAP. thanks!
As of today I am.
do you have an update on this? Now that the jobs are set up, this is a blocker for 1.6 upgrade testing. Thanks!
I started looking at this yesterday, figured out what was going wrong. The problem is due to enabling RBAC. There are a couple options to fix. I should have a fix prepared and out for review today or tomorrow.
With the I hit another problem. Here is the apiserver error log: Reopen this.
from release-1.5 to master, I am not able to reproduce. verified that this fixed our CI tests.
Sorry, I forgot that my script is upgrade from 1.5.3 to 1.6.0-beta.1, which doesn't include your fix. Will verify again and close this one if it works. :)
Wait, it seems that you only changed , then I'm indeed using the latest to upgrade from v1.5.3 to v1.6.0-beta.1. Is there anything missing in v1.6.0-beta.1?
I think beta.2 rolled up this fix.

master-only upgrades seem to be working correctly now, however full-cluster upgrades aren't getting the cluster into a functioning state: <*errors.errorString | 0xc422c96f10: { I0309 08:45:31.969] s: "failed to get logs from downwardapi-volume--04e7-11e7-986c- for client-container: the server has asked for the client to provide credentials (get pods downwardapi-volume--04e7-11e7-986c-)", See https://k8s-
OK. It works from v1.5.3 to v1.6.0-beta.2. Close this one. :)
looking.... sounds like it might be related :( :(
broke upgrades since it adds a group to certificates that certificates generated with 1.5 don't have. This is causing:
upgrades should still have the existing ABAC policy in place
ahh, ok. I'll look into why that's not helping.
Infinity bugs. PTAL at
can this be closed?


Question: makeWindowsHostPod(nil), "spec.ephemeralContainers[0].securityContext.windowsOptions.hostProcess: Invalid value: false: pod hostProcess value must be identical", }, <ins> { "Add ephemeral container to static pod", func() *core.Pod { p := makePod(nil) p.Spec.NodeName = "some-name" p.ObjectMeta.Annotations = map[string]string{ core.MirrorPodAnnotationKey: "foo", } p.Spec.EphemeralContainers = []core.EphemeralContainer{{ EphemeralContainerCommon: core.EphemeralContainerCommon{ Name:                     "debugger1", Image:                    "debian", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }} return p }(), func() *core.Pod { p := makePod(nil) p.Spec.NodeName = "some-name" p.ObjectMeta.Annotations = map[string]string{ core.MirrorPodAnnotationKey: "foo", } return p }(), "Forbidden: static pods do not support ephemeral containers", }, </ins> } for _, tc := range tests {

I created an ephemeral container for a Static Pod using the command. The command succeed. The API server accepted the ephemeral container: However, the container is not created. Its status is also not available: I expected one of these: (a) the API server rejects the request to create an ephemeral container for a Static Pod (b) kubelet creates the ephemeral container, and update the Static Pod's status sub-resource Using a recent version of , v0.17 or newer, and v1.25.0 or newer, create a cluster and create an ephemeral container running . Since the ephemeral container is not running, the output of is 0. The does not address Static Pods. <details</details<detailsNone </details<details</details<details</details<details docker exec -it kind-control-plane containerd --version containerd v1.6.9 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The umbrella issue for ephemeral containers is
/cc
Control plane machines provisioned using run control plane components (etcd, kube-apiserver, kube-controller-manager, kube-scheduler) as Static Pods. Being able to create ephemeral containers for these Static Pods in order to debug the running processes is a use case worthy of mention. If ephemeral containers are, by design, not intended to be used with Static Pods, let's please justify why that is the case, and try to document an alternative.
Thank you for raising this issue. Indeed we either need to update docs or fix it. Can you please provide the kubelet log? We discussed this on triage meeting and not sure if this is actually supported or not. Kubelet log may give us some information quickly in case it is some bug. /triage needs-information /cc
Thanks for triaging it quickly! Yes, I'll do that.

(I didn't do it initially, because even after I bumped kubelet log verbosity to 10, I found no references of the ephemeral container names in the log. I'll just attach the complete kubelet log here.)
Attached is the (compressed) kubelet log running in the kind-control-plane container. I bumped up its log verbosity to 8 prior to creating the ephemeral container. I created the ephemeral container using the command described above at 18:42:31.
hi I was not able to reproduce this. I'm using kind 0.17 and kubectl 1.25.4. Can you try upgrading kubectl? I'd like to know if its some specific version. I looked through the kubelet log and didn't see any signs of trouble. Here's the data from my test run:
Thanks for trying to reproduce this. Your results are consistent with mine. What I am reporting is that, despite having the correct spec, the actual container is not created. However, I failed to explain how to verify that claim. We could verify it, for example, by listing all the processes running in the container. We should find the process, since it would be running in the ephemeral container. In my case, the process is not present: Moreover, since the ephemeral container has a shell executable, I should be able to run . In my case, the command fails: I realize that the is not highly reliable; might not work for other reasons, even if the ephemeral container is created.
I do see a difference in the test results, but I still can't exec into the ephemeral container. I'll run some more tests. Thanks Here is the pod spec showing something for the ephemeral container, but i can't exec into it anyway:
We should not allow ephemeral containers for static pods… the API content is not authoritative for static pods, the manifests on disk are, so I would expect the kubelet to ignore API spec content and the API to disallow adding ephemeral containers to static pods
It sounds like a documentation update would be appropriate here. I think the Static Pods page would be a good place for it. I'll get a PR going. Thanks
/remove-kind bug /kind documentation /sig docs
/remove-triage needs-information Is this good to close?

We could additionally document why you can't add an ephemeral container (which is that mirror pods are essentially read-only).
If the /ephemeralcontainers subresource doesn't currently reject attempts to add to a static pod, I think it should, to make it clearer the addition won't be effective
Does the API server actually know that mirror pods are mirror pods? I wonder how you can tell.
it seems it can if it wants
I think that is just updating this function to fail if the annotation is set, right? ... and add tests, of course can you confirm?
yeah, that's right
any taker on this one?
/kind bug
I'll take it, per the discussion here: add a check in to reject the attempt to add an ephemeral container if the request is aimed at a static pod. Also include a test case. Also some additional documantation comments from this ticket and from kubernetes/website
/assign Thanks


Question: }, }, { <del> description: "pod specified toleration for taint `unreachable`, expect add toleration for `notReady:NoExecute`", </del> <ins> description: "pod specified toleration for taint `unreachable`, expect add toleration for `not-ready:NoExecute`", </ins> requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: import ( "sync" <del> "k8s.io/component-base/metrics" </del> "k8s.io/component-base/metrics/legacyregistry" <del> ) const kubeProxySubsystem = "kubeproxy" var ( SyncProxyRulesLatency = metrics.NewHistogram( &metrics.HistogramOpts{ Subsystem:      kubeProxySubsystem, Name:           "sync_proxy_rules_duration_seconds", Help:           "SyncProxyRules latency in seconds", Buckets:        metrics.ExponentialBuckets(0.001, 2, 15), StabilityLevel: metrics.ALPHA, }, ) // SyncProxyRulesLastTimestamp is the timestamp proxy rules were last // successfully synced. SyncProxyRulesLastTimestamp = metrics.NewGauge( &metrics.GaugeOpts{ Subsystem:      kubeProxySubsystem, Name:           "sync_proxy_rules_last_timestamp_seconds", Help:           "The last time proxy rules were successfully synced", StabilityLevel: metrics.ALPHA, }, ) </del> <ins> "k8s.io/kubernetes/pkg/proxy/metrics" </ins> ) var registerMetricsOnce sync.Once <ins> // RegisterMetrics registers kube-proxy metrics for Windows modes. </ins> func RegisterMetrics() { registerMetricsOnce.Do(func() { <del> legacyregistry.MustRegister(SyncProxyRulesLatency) legacyregistry.MustRegister(SyncProxyRulesLastTimestamp) </del> <ins> legacyregistry.MustRegister(metrics.SyncProxyRulesLatency) legacyregistry.MustRegister(metrics.SyncProxyRulesLastTimestamp) legacyregistry.MustRegister(metrics.EndpointChangesPending) legacyregistry.MustRegister(metrics.EndpointChangesTotal) legacyregistry.MustRegister(metrics.ServiceChangesPending) legacyregistry.MustRegister(metrics.ServiceChangesTotal) legacyregistry.MustRegister(metrics.SyncProxyRulesLastQueuedTimestamp) </ins> }) }

function on is not being used, and kproxy metrics are not showing on endpoint. See kube-proxy related metrics. Access kube-proxy on and search for kube-proxy specific metrics. No response <detailsKube-proxy build from master </details<details</details<details</details<details</details<details</details<details</details
/sig windows
/triage accepted /priority important-soon
/milestone v1.24