Question: } } <ins> func TestListPlugins(t *testing.T) { pluginPath, _ := filepath.Abs("./testdata") expectPlugins := []string{ filepath.Join(pluginPath, "kubectl-foo"), filepath.Join(pluginPath, "kubectl-version"), } verifier := newFakePluginPathVerifier() ioStreams, _, _, _ := genericclioptions.NewTestIOStreams() pluginPaths := []string{pluginPath} o := &PluginListOptions{ Verifier:  verifier, IOStreams: ioStreams, PluginPaths: pluginPaths, } plugins, errs := o.ListPlugins() if len(errs) > 0 { t.Fatalf("unexpected errors: %v", errs) } if !reflect.DeepEqual(expectPlugins, plugins) { t.Fatalf("saw unexpected plugins. Expecting %v, got %v", expectPlugins, plugins) } } </ins> type duplicatePathError struct { path string }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.


Question: reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: dv.Set(sv.Convert(dt)) return nil <ins> case reflect.Float32, reflect.Float64: dv.Set(sv.Convert(dt)) return nil </ins> } case reflect.Float32, reflect.Float64: switch dt.Kind() {

What happened: When converting JSON representations of v1beta1 CustomResourceDefintions with fields set to a whole number, the function returns 'cannot convert int64 to float64'. Making the field decimal numbers converts as expected. What you expected to happen: OpenAPI schema should convert from unstructured.Unstructured whether whole or decimal representation. How to reproduce it (as minimally and precisely as possible): A Go test with a minimal CRD & failing test is at The repo includes a with pinned versions of dependencies. Use to run it and get a failure. Anything else we need to know?: This was identified by someone reviewing a that waited on CRDs to be 'ready' before restoring CRDs from their serialized JSON representations. The reproduced case is based on his sample CRD. There's also a Helm 2 issue referencing this, with an included fix. I believe may be related, but I'm not 100% sure. Environment: Kubernetes version (use ): apimachinery 0.17.2
/sig api-machinery
I don't think number type conversion is intended to be performed by that helper. It's primary purpose is to round-trip between a typed and unstructured object without requiring json serialization/deserialization. If you have JSON you want to unmarshal, you can do so directly into the CRD go type without first unmarshaling into an Unstructured object.
/assign
Apologies for the delayed response. In this was actually with information returned by the Kubernetes API server, which I then tried to change into the CRD Go type. The original goal was for Velero (a backup/restore tool) to check a CRD in the cluster for readiness before restoring any CRs. In Velero's case, because we deal with more or less any object that could be in the API server, we're using a Dynamic client as we can't import every possible Go type, though well-known ones like CRDs are different. The example I have is the simplest possible reproduction case I could come up with, without involving a full Kubernetes API server. Given we're constrained to using Unstructured types from the Kubernetes API server, is there an alternative method we could use for translating them into the CRD Go types?
I'm not sure I understand. If you're working with go types, you can unmarshal into the typed structs, correct?

Not always, because in some instances, we hit this error. We got another report of this today (), which I traced back to my code at , the call to take the Unstructured data to the Go type. I should clarify - Velero's not working with JSON directly at this point. It's working with .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle-stale This is still affecting Velero, though we've worked around it by not using on CRDs returned directly from the Kube API server as Unstructured through a dynamic client. We instead marshal to JSON and back for now. I think I may have not explained my case clearly - is there anything I can do or provide to help resolve this?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
apologies for the delay, I dug into this and opened as a fix
Thanks a ton!


Question: - name: vol mountPath: /scrub EOF <ins> # fixup the alternate registry if specified if [[ -n "${KUBE_ADDON_REGISTRY:-}" ]]; then sed -i -e "s@k8s.gcr.io@${KUBE_ADDON_REGISTRY}@g" "${PV_RECYCLER_OVERRIDE_TEMPLATE}" fi </ins> } function wait-till-apiserver-ready() {

Noticed this issue when looking at logs to figure out which images we use in a e2e test and from where. for example: Looks like it is from here:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /sig storage cc
see background in and
slack discussion -
Looks like we can switch to see below:
looks like this image supports all the arch(es) we care about as well: !


Question: <del> //go:build go1.14 // +build go1.14 /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package flag import ( "crypto/tls" ) func init() { // support official IANA names as well on go1.14 ciphers["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256"] = tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 ciphers["TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"] = tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 } </del>

Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,


Question: }, wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict), }, <ins> { name: "pv with beta label,node with ga label, matched", Pod:  createPodWithVolume("pod_1", "Vol_1", "PVC_1"), Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "host1", Labels: map[string]string{ v1.LabelTopologyZone: "us-west1-a", }, }, }, }, { name: "pv with beta label,node with ga label, don't match", Pod:  createPodWithVolume("pod_1", "vol_1", "PVC_1"), Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "host1", Labels: map[string]string{ v1.LabelTopologyZone: "us-west1-b", }, }, }, wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict), }, </ins> } for _, test := range tests {

In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue?

If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: package apimachinery import ( <ins> "crypto/rand" </ins> "crypto/x509" "encoding/json" <ins> "fmt" </ins> "io/ioutil" <ins> "math/big" </ins> "os" "strings" "time"

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: _, err = f.ClientSet.AppsV1().ReplicaSets(ns).Patch(context.TODO(), rsName, types.StrategicMergePatchType, []byte(rsPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch ReplicaSet") <del> ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) </ins> defer cancel() _, err = watchtools.Until(ctx, rsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if rset, ok := event.Object.(*appsv1.ReplicaSet); ok {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Since March 25th 2021, mostly windows jobs but a few others: 22 ci-kubernetes-e2e-windows-gce-2004 25 ci-kubernetes-e2e-windows-gce-20h2 17 ci-kubernetes-e2e-windows-containerd-gce 12 ci-kubernetes-e2e-windows-gce-2019 10 ci-kubernetes-kind-conformance-parallel-ipv6 5 ci-kubernetes-kind-e2e-parallel 4 ci-kubernetes-e2e-gci-gce-alpha-enabled-default Kubernetes e2e suite.[sig-apps] ReplicaSet Replace and Patch tests [Conformance] links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig windows /sig apps /kind flake
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's started failing consistently in Windows Dockershim
looks like there were updates to this tests which may have caused issues for Windows: was merged on the 24th and we started seeing flakes the day after
a 30 second time out that is running waiting for the pods to come up. from The windows nodes are still pulling the image at the 30 second timeout. Test failes at 10.11.31: You can see the image is still being pulled.
the pod for was finished pulling but hadn't started:
same for
It is definitely a timing issue for Windows (due to slower pull times) and looks to be a timing issue even on linux though it happens fewer times. We could increase the time out... I noticed it doesn't look like we have a consistent timeout for pods starting across tests.
Another option would be to use the which has a longer timeout (5 min) and is used earlier in the tests. Once running check the values on the replica set
/milestone v1.21


Question: Requests: getNodeCPUAndMemoryCapacity(f), }) <ins> _, err = f.ClientSet.SchedulingV1().PriorityClasses().Create(systemCriticalPriority) gomega.Expect(err == nil || errors.IsAlreadyExists(err)).To(gomega.BeTrue(), "failed to create PriorityClasses with an error: %v", err) </ins> // Create pods, starting with non-critical so that the critical preempts the other pods. f.PodClient().CreateBatch([]*v1.Pod{nonCriticalBestEffort, nonCriticalBurstable, nonCriticalGuaranteed}) f.PodClientNS(kubeapi.NamespaceSystem).CreateSyncInNamespace(criticalPod, kubeapi.NamespaceSystem)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: return framework.NewStatus(framework.Error, err.Error()) } node := nodeInfo.Node() <del> // Debugging purpose: print the score for each node. // Score must be a pointer here, otherwise it's always 0. if klog.V(10) { defer func(score *int64, nodeName string) { klog.Infof("%v -> %v: PodTopologySpread NormalizeScore, Score: (%d)", pod.Name, nodeName, *score) }(&scores[i].Score, node.Name) } </del> if maxMinDiff == 0 { scores[i].Score = framework.MaxNodeScore

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Logging getting points for ALL priorities set in the default scheduler policy Why is this needed: We have Kubernetes v1.14 with a problem of planning pods for three nodes. The problem is that the scheduler unevenly distributes the pods across the nodes. For example, the CPU on two nodes Node02 and Node03 are 50% loaded, while the CPU on the third Node01 is not loaded at all. While that the pods continue to be planned on Node02 and Node03, but they are not planned on Node01 at all. Scheduler logs (maximum logging level) with uniform planning: and uneven planning: As you can see, the sum of points for the priorities BalancedResourceAllocation, LeastResourceAllocation, SelectorSpreadPriority, TaintTolerationPriority is the same in both cases, but the final result is different. There is no way to understand for what specific policy priorities Node01 did not get 3 points, for to be on a par with the others, because the set of points of some priorities is not logged.
/sig scheduling
I think maybe we missed some logs which can reflect the situation of giving the score. WDYT? If you confirmed, I want to handle this issue.
Thank you for the answer! Here is the full scheduling log (logging level 10): maybe I’m really missing something, but I don’t see information about other priorities here.
Missing scores may be due to some priority functions are using non-map-reduce style, so they didn't hit the printing logic. We can add a logging function there to help you better debugging it. However, the code gets refactored in recent releases, and 1.14 is out of our supporting cycles (only 1.15, ,1.16, 1.17). Can you upgrade to a newer version, or I can provide a code snippet for you to compile on your local 1.14 branch?
Thank you, but I already solved my problem by creating an additional scheduler with a custom scheduling policies for my cronjob. However, it would be cool if you the logging function in the supported versions :)
SG. I created to address this.
I'm running v1.23.8, and I've tried and I'm still not seeing any of the scoring output you in Am I doing something wrong?

Shouldn't this be available in v1.23?
is it possible there is only 1 node candidate after filtering? In that case, the scoring logic will be bypassed as it's meaningless to score for 1 node. If it's not the case, could you check the beginning of the scheduler log, to ensure is configured properly?


Question: visible to subsequent reads. release: v1.22 file: test/e2e/apps/statefulset.go <ins> - testname: Conformance tests minimum number of nodes. codename: '[sig-architecture] Conformance Tests should have at least two untainted nodes [Conformance]' description: Conformance tests requires at least two untainted nodes where pods can be scheduled. release: v1.23 file: test/e2e/architecture/conformance.go </ins> - testname: CertificateSigningRequest API codename: '[sig-auth] Certificates API [Privileged:ClusterAdmin] should support CSR API operations [Conformance]'

I can create a cluster with one node and pass conformance, ~there are some tests that require at least 2 nodes to run~ ~~ ~This is because the tests Skip if they are not able to run~ ~~ ~As consequence, the list of tests for conformance depends on the number of nodes, a single cluster node (that is not a cluster by definition of cluster :)) will run less tests than a multi cluster node and still be Conformant~ EDIT I have to correct myself, it seems that the conformance tests doesn't skip, the problem is: there are no conformance tests that exercise anything that requires more than one node, hence, single node cluster can pass conformance. I expect a cluster (a group of similar things or people positioned or occurring closely together) and a "distributed system" to not be able to be single-node. In addition, lots of most important Kubernetes features can not be exercised on one node, lowering the quality bar. local- pass conformance Related discussion <details</details<details</details<details</details<details</details<details</details<details</details
/assign Maybe this has changed, but last time I've checked the minimum for conformance was 1 control-plane and 2 workers, if that didn't change I will submit a PR to enforce it in the suite Regarding sig-network, with one node you can not verify the networking model, where pod to pod in different nodes should be able to communicate without NAT, that is a strong requirement ... I have to review closely what tests do we have in Conformance
/area conformance /sig network
that seems pretty intentional... changing the definition of conformance to require multiple nodes probably deserves more feedback than an issue/PR
I'm not changing the definition, just asserting it
if we are testing features like replicaset, daemonset, services, ..., in one node only we are missing a lot of testing on those features ... a podman/docker single node can be a kubernetes conformance cluster ... do I need a scheduler or network for anything, I'm just deploying always in the same node?
No I don't think so. Where are your results and configuration? You need two schedulable nodes to pass the conformance suite. That's not correct, we test rolling pod behavior with replicasets. This test requires multiple nodes. There is at least one.

That job is not passing conformance, from the configuration: (see: ) Conformance requires not setting --, you must run all tests without skipping. We don't, at the very least replicaset is being tested multi-node, just not in this job that is not running the full conformance suite. When conformance results are submitted they are now validated to ensure that all tests were run.


Question: t.Errorf("Expected to get mid prority pod, got: %v", p) } } <ins> // TestHighProirotyFlushUnschedulableQLeftover tests that pods will be moved to // activeQ after one minutes if it is in unschedulableQ func TestHighProirotyFlushUnschedulableQLeftover(t *testing.T) { q := NewPriorityQueue(nil) midPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-midpod", Namespace: "ns1", UID:       types.UID("tp-mid"), }, Spec: v1.PodSpec{ Priority: &midPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } highPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-highpod", Namespace: "ns1", UID:       types.UID("tp-high"), }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } q.unschedulableQ.addOrUpdate(&highPod) q.unschedulableQ.addOrUpdate(&midPod) // Update pod condition to highPod. podutil.UpdatePodCondition(&highPod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Time{Time: time.Now().Add(-1 * unschedulableQTimeInterval)}, }) // Update pod condition to midPod. podutil.UpdatePodCondition(&midPod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Time{Time: time.Now().Add(-1 * unschedulableQTimeInterval)}, }) if p, err := q.Pop(); err != nil || p != &highPod { t.Errorf("Expected: %v after Pop, but got: %v", highPriorityPod.Name, p.Name) } if p, err := q.Pop(); err != nil || p != &midPod { t.Errorf("Expected: %v after Pop, but got: %v", medPriorityPod.Name, p.Name) } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scheduler places unschedulable pods in "unschedulabe" queue and retries them only when certain events happen that could potentially make them schedulable. This logic works well in almost all scenarios, but inevitable race condition in large distributed systems, could potentially cause some events to be seen before pods are to the unschedulable queue. If this happens, pods may be left in the unschedulable queue and not be retried. Such scenarios should be rare and even if they occur, usually there are other events that trigger a retry and cover them. However, if such scenarios happen in smaller and low churn clusters, other events may not be seen for a while and pods may be stuck in the unschedulable queue for a long time. The scheduling queue () has a method that is triggered every second to move backed off pods to the active queue. We should add another go routine that is triggered every 30 seconds and moves pods whose last schedule time is older than 1 minute from "unschedulble" queue to the active queue. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug I don't know if this should be marked as a bug. I don't have any report of such issue. It is more of a precaution to prevent issues. /sig scheduling
I am worried if this will lead to continuous repeated scheduling failures?
What is the problem with repeated scheduling failures? The scheduler had a FIFO until recently. It used to keep retrying unschedulable pods all the time.
can I take this?
Sure, thanks!
GitHub didn't allow me to assign the following users: denkensk. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It increases the schedule possibility of unschedulble pods and the chances of scheduling failures. Should we also consider latter?


Question: return b.backoff.Step() } <del> // Backoff implements BackoffManager.Backoff, it returns a timer so caller can block on the timer for backoff. </del> <ins> // Backoff implements BackoffManager.Backoff, it returns a timer so caller can block on the timer for exponential backoff. // The returned timer must be drained before calling Backoff() the second time </ins> func (b *exponentialBackoffManagerImpl) Backoff() clock.Timer { <del> b.backoffTimer.Reset(b.getNextBackoff()) </del> <ins> if b.backoffTimer == nil { b.backoffTimer = b.clock.NewTimer(b.getNextBackoff()) } else { b.backoffTimer.Reset(b.getNextBackoff()) } </ins> return b.backoffTimer }

I noticed that this code: was printing which should not happen. Also occurs in wait.Forever. I suspect that in regressed this as that was the most recent major update and I only started seeing this on a recent apimachinery bump.
/sig api-machinery /assign as the reviewer
GitHub didn't allow me to assign the following users: zhan849. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery /assign as the reviewer Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
This is a correctness regression because the wait methods expect not to be run run more often than the period and may guard expensive behavior. Suggest a revert followed by a more rigorous test and review on merge.
thanks for reporting this, and I apologize about the regression I made. I did some quick test, and I am able to reproduce it occasionally, which indicate a race somewhere: Test: Failed output: Successful output? In the original code I checked in, I initialized the timer in the backoff manager construction time: However, after I changed to initializing the timer when is called, I'm not able to reproduce the race. The results indicated that the timer initialization is likely to be the place that caused the regression , and seems to be able to get fixed by change the way the timer gets initialized. However I'm unable to identify why it could cause such race the first time. Are you able to provide any insights? /cc
why are we initializing a timer with a zero duration? that looks related to this issue:
I see us resetting a timer we have no guarantee is stopped/expired:
dropped the information needed to accurately stop/reset the timer as documented
Hi thanks for providing the information! seems like the correct place to initialize the timer is in the "Backoff" method, since zero duration is problematic.

When I was putting up the interface, I documented that the timer will not be drained, and the reason is that returns a timer explicitly for user to block on that, so it might be confusing to block in itself Shall we change the impl to always drain the channel in backoff instead or relying on user to use it correctly? with this being said, will be changed to , which means that the will still rely on user's providing information about whether the channel is drained or not.
/cc thoughts? ^^
- sorry for that - we hit the bug with resetting timer not the first time, I should have caught that (I remember at least two other bugs with that). I'm not sure revert is a good option, because there were PRs that were chained on that: - this would have to be a very careful manual revert which I'm not sure it justifies the risk, given that 1.18 is already out. Currently the behavior is unpredictable, because if getNextbackoof() called for the first time is non-zero, it may or may not be ready immediately depending on the race. That is obviously bad. Initializing with infinity may solve this particular problem, but I need to think more about it to ensure that we don't have any problems in the later phase too...
thanks ! and sry again I did not read the timer's doc carefully enough but simply followed the Clock interface :( To fix, we can also initialize the timer to nil and create a fresh one when is first called, it's safe as this component is designed to run in single-threaded environment. But to ensure timer is drained before reset, we need to have user explicitly pass in "if timer is drained" into the to accurately decide if needs to drain. Or, is it ok to document the behavior and assume user will drain on before calling next ?
Very quick (I might have missed something) glance from the code suggests, that the only place we would need to change is actually BackoffUntil method. So passing the "if timer is drained" bit seems less error prone imho.
We should avoid changing the interface in a patch release.

I think initializing the timer in Backoff() and documenting the returned timer must be expired and drained before calling Backoff() again is the most compatible thing we can do:
Thanks and I'm also more inclined to Jordan's proposal, besides avoid interface change, if user are able to accurately pass "if timer is drained" info, it's fair to let user finish draining. do you have a PR already? of if not, I can make one, and add more tests if possible about this failure case.
I do not have a PR open


Question: runServiceTest, runAPIVersionsTest, runMasterServiceTest, <ins> runSelfLinkTest, </ins> } var wg sync.WaitGroup wg.Add(len(testFuncs))

Had a report of this over email. Filing this issue so we don't forget to investigate.
Can't reproduce.


Question: return } ref := &v1.ObjectReference{ <del> Kind:      "Pod", Name:      nsName.Name, Namespace: nsName.Namespace, </del> <ins> APIVersion: "v1", Kind:       "Pod", Name:       nsName.Name, Namespace:  nsName.Namespace, </ins> } tc.recorder.Eventf(ref, v1.EventTypeNormal, "TaintManagerEviction", "Cancelling deletion of Pod %s", nsName.String()) }

Events from node taint manager doesn't report involved object apiVersion: ! apiVersion is reported as other events. For example: ! a pod on node A (not master) off node A is reported, but without apiVersion in involvedObject. The apiVersion is not reported in The issue is still not fixed in master: ! <details</details<detailsIrrelevant </details<details</details<details</details<details</details<details</details
/sig node /sig apps
/assign In the latest version, the issue is confirmed
one suggestion you should wait for the corresponding SIG to triage the issue and accept it before sending PRs, the chances of your PR getting reviewed and approver are much higher
You are right sorry about that. It won’t happen again.
no no worries, it was a suggestion, is not that you did something wrong, but it is possible that your PR doesn't get reviews and you get frustrated
/remove-sig node please re-add if this is node-related issue
I think it is a node related issue. The root cause in taint manager.
/sig node
/assign /triage accepted


Question: MinExternalEtcdVersion = "3.2.18" // DefaultEtcdVersion indicates the default etcd version that kubeadm uses <del> DefaultEtcdVersion = "3.5.3-0" </del> <ins> DefaultEtcdVersion = "3.5.4-0" </ins> // Etcd defines variable used internally when referring to etcd component Etcd = "etcd"

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: proxier.endpointsSynced = true } <ins> func deleteEndpoints(proxier *Proxier, allEndpoints ...*v1.Endpoints) { for i := range allEndpoints { proxier.OnEndpointsDelete(allEndpoints[i]) } proxier.mu.Lock() defer proxier.mu.Unlock() proxier.endpointsSynced = true } </ins> func makeTestEndpoints(namespace, name string, eptFunc func(*v1.Endpoints)) *v1.Endpoints { ept := &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Two services referring the same endpoint. One of the services was deleted. This brought down the connectivity of the other service. What you expected to happen: The connectivity to service should continue to work How to reproduce it (as minimally and precisely as possible): two services referencing same endpoint one of the services connectivity is broken to the other service Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Do you mean a service referencing the same pod, or literally the same Endpoints object?
I meant service referencing the same pod.
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Is this still an issue? If so, it would be helpful to get an answer to Jordan's question above. If this Service is referencing the same Endpoints object that would be a very different issue than if it was referencing the same Pod. /triage needs-information
This issue happens when multiple Services are referencing a same POD. I mean the same backend IP address but different ports. i.e. the POD is hosting two workloads on different ports.
/sig windows
Is this only observed in windows environments?
Yes this happens only on Windows environments
Thanks can you provide more information about the version of Kubernetes you're using? Do you know what proxier mode kube-proxy is running with? Maybe winkernel? I'm not as familiar with the Windows implementation here, adding to help debug this.


Question: // TODO: get the real minion object of ourself, // and use the real minion name and UID. ref := &api.ObjectReference{ <del> Kind: "Minion", Name: kl.hostname, UID:  kl.hostname, </del> <ins> Kind:      "Minion", Name:      kl.hostname, UID:       kl.hostname, Namespace: api.NamespaceDefault, </ins> } record.Eventf(ref, "", "starting", "Starting kubelet.") }

From kubelet logs: E1117 23:17:17. ] Sleeping: Unable to write event: event "e2e-test-etune-minion-1.c.seventh-circle-487.internal." is invalid: involvedObject.namespace: invalid value '' The kubelet code says: I'll update when I figure out where the record.Event is coming from. The code is:
Event in question is: I1117 23:25:42. ] Event(api.ObjectReference{Kind:"Minion", Namespace:"", Name:"e2e-test-etune-minion-1.c.seventh-circle-487.internal", UID:"e2e-test-etune-minion-1.c.seventh-circle-487.internal", APIVersion:"", ResourceVersion:"", FieldPath:""}): status: '', reason: 'starting' Starting kubelet.
I think the namespace needs to be kubelet. Note that one bad event blocks other good events from getting sent. Probably better to change that.
I'll file a new one for bad event blocking good.
I have a similar issue related to event. When generating event for image, what kind of namespace I should use? Image is not APIObject kubernetes managed, and it could shared crossing several containers even belongs to different pod. Currently I plan to just use default when creating ObjectReference for image related event. If we use kubelet for event related to minion, are we going to make it reserved one?
what events are you planning to add for images? "default" seems like a good enough namespace for minion events for now.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/resource" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/fieldpath" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/types"

A "kubectl describe" on a pod ID give a lot of informations about that pod but don't you think it should also print the pod's env var ?


Question: if isClosedError(err) { return } <del> if info, exists := proxier.getServiceInfo(service); !exists || info != myInfo { </del> <ins> if !myInfo.isAlive() { </ins> // Then the service port was just closed so the accept failure is to be expected. return }

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: "testing" "github.com/Azure/azure-sdk-for-go/services/containerregistry/mgmt/2019-05-01/containerregistry" <ins> "github.com/Azure/go-autorest/autorest/azure" </ins> "github.com/Azure/go-autorest/autorest/to" )

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: func (cont *GCEIngressController) CreateStaticIP(name string) string { gceCloud := cont.Cloud.Provider.(*gcecloud.GCECloud) addr := &compute.Address{Name: name} <del> ip, err := gceCloud.ReserveGlobalAddress(addr) if err != nil { </del> <ins> if err := gceCloud.ReserveGlobalAddress(addr); err != nil { </ins> if delErr := gceCloud.DeleteGlobalAddress(name); delErr != nil { if cont.isHTTPErrorCode(delErr, http.StatusNotFound) { Logf("Static ip with name %v was not allocated, nothing to delete", name)

/kind bug What happened: Services of type LoadBalancer (GCE) with IP addresses promoted to static (by user or by orphan) cause the service controller to panic. The constant panicking will cause a reboot loop of the pod. How to reproduce it (as minimally and precisely as possible): a service on GCP it the GCP Console, promote the IP address of the LB to static and set the name to the loadBalancerName which is the same name as the forwarding rule. Environment: Kubernetes version: 1.7.0 AND 1.7.1 Cloud provider or hardware configuration: GCE Symptoms -controller-manager crash loops on the master. Mitigation** Option 1: Build your own version of branch for the master. Only the kube-controller-manager image needs to be updated. Option 2: For all affected services, set the field to the known external IP address. Steps: list of reserved IPs gcloud compute addresses list NAME REGION ADDRESS STATUS us-central1 35.35.35.35 INUSE list of all Services of type LoadBalancer kubectl get svc --all-namespaces | grep "LoadBalancer" default my-service LoadBalancer 10.0.99.201 35.35.35.35 80:/TCP,443:/TCP 58m each service of type LoadBalancer with an IP that's reserved (in the list above), continue to next step. the service's field equal to the known IP. Example_ apiVersion: v1 kind: Service metadata: creationTimestamp: 2017-07-13T00:09:31Z labels: app: echo name: my-service namespace: default resourceVersion: "" selfLink: /api/v1/namespaces/default/services/my-service uid: -675f-11e7-bee2- spec: clusterIP: 10.0.99.201 externalTrafficPolicy: Cluster loadBalancerIP: 35.35.35.35 <------- Field set to below. ports: name: port-1 nodePort: port: 80 protocol: TCP targetPort: 80 name: port-2 nodePort: port: 443 protocol: TCP targetPort: 443 selector: app: echo sessionAffinity: ClientIP type: LoadBalancer status: loadBalancer: ingress: ip: 35.35.35.35
/sig network /assign /area platform/gce
cc


Question: # By default, the latest image from the image family will be used unless an # explicit image will be set. GCI_VERSION=${KUBE_GCI_VERSION:-} <del> IMAGE_FAMILY=${KUBE_IMAGE_FAMILY:-cos-97-lts} </del> <ins> IMAGE_FAMILY=${KUBE_IMAGE_FAMILY:-cos-109-lts} </ins> export MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} export MASTER_IMAGE_FAMILY=${KUBE_GCE_MASTER_IMAGE_FAMILY:-${IMAGE_FAMILY}} export MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud}

These tests are completely failing due to a compilation issue so no tests are running. NA You can go to the containerd logs in artifacts and see that GLIBC_2.34 is not found. Typically these fixes can be resolved by an image update. These tests don't seem to point to any default image. /sig node
One of these is not like the other! Looks like ci-cos-cgroupv2-containerd-node-e2e is failing due to a missing clone of containerd. fixes ci-cos-cgroupv2-containerd-node-e2e.
/assign I noticed that the jobs were passing with image , started failing with image . I've filed an internal support case, will post more updates later.
/triage accepted /priority important-soon
COS team came back to us, it seems that the change is in how containerd is compiled (lGLIBC versions are the same across both images). I'll take a look at this issue next week, as I'm oncall this one.
This PR fixed this issue. I striked it out on this issue.
Added to the list as this is also failing for a similar reason.
Containerd can be built statically to avoid this kind of issues: ~/git/containerd (1.7.7) $ ldd bin/containerd not a dynamic executable
We don't even need to statically rebuild containerd to fix this as statically built binaries are published by the containerd project as part of their releases, e.g.
The binaries that we use in testing are from the main branch of containerd. So getting a static binary from the release is not an option.
What is the glibc version used in ? The reason of failure may be because the  image that is used for building containerd switched from bullseye to bookworm which has
Correct. However, we can still statically build containerd binaries or ask containerd maintainers to publish prebuilt static binaries from their main branch. With dynamically built executables we're going to have this kind of issues in future.
/reopen Looks like did not fix this so reopening the issue.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
second attempt to fix it:
Nice work
Thank you. is not triggered yet. Next run will be tomorrow. I hope it's fixed.


Question: "k8s.io/kubernetes/pkg/util/system" taintutils "k8s.io/kubernetes/pkg/util/taints" utilversion "k8s.io/kubernetes/pkg/util/version" <del> "fmt" "sync" "time" "github.com/golang/glog" </del> ) func init() {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: If TaintNodesByCondition enabled, we should disable and predicates in scheduler. In TaintNodeByCondition, we will handle those conditions: , , and ; NetworkUnavailable checking is removed by predicates, but and not (no predicate for ). /cc
/sig scheduling /sig node
/assign


Question: "!", "--ctstate", "RELATED,ESTABLISHED,DNAT", "-j", "DROP"); err != nil { klog.Errorf("Failed to ensure rule to drop invalid localhost packets in %v chain %v: %v", utiliptables.TableFilter, KubeFirewallChain, err) <del> return </del> <ins> return false </ins> } } if _, err := iptClient.EnsureRule(utiliptables.Prepend, utiliptables.TableFilter, utiliptables.ChainOutput, "-j", string(KubeFirewallChain)); err != nil { klog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableFilter, utiliptables.ChainOutput, KubeFirewallChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureRule(utiliptables.Prepend, utiliptables.TableFilter, utiliptables.ChainInput, "-j", string(KubeFirewallChain)); err != nil { klog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableFilter, utiliptables.ChainInput, KubeFirewallChain, err) <del> return </del> <ins> return false </ins> } // Setup KUBE-MARK-MASQ rules masqueradeMark := getIPTablesMark(kl.iptablesMasqueradeBit) if _, err := iptClient.EnsureChain(utiliptables.TableNAT, KubeMarkMasqChain); err != nil { klog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableNAT, KubeMarkMasqChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureChain(utiliptables.TableNAT, KubePostroutingChain); err != nil { klog.Errorf("Failed to ensure that %s chain %s exists: %v", utiliptables.TableNAT, KubePostroutingChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureRule(utiliptables.Append, utiliptables.TableNAT, KubeMarkMasqChain, "-j", "MARK", "--or-mark", masqueradeMark); err != nil { klog.Errorf("Failed to ensure marking rule for %v: %v", KubeMarkMasqChain, err) <del> return </del> <ins> return false </ins> } if _, err := iptClient.EnsureRule(utiliptables.Prepend, utiliptables.TableNAT, utiliptables.ChainPostrouting, "-m", "comment", "--comment", "kubernetes postrouting rules", "-j", string(KubePostroutingChain)); err != nil { klog.Errorf("Failed to ensure that %s chain %s jumps to %s: %v", utiliptables.TableNAT, utiliptables.ChainPostrouting, KubePostroutingChain, err) <del> return </del> <ins> return false </ins> } // Set up KUBE-POSTROUTING to unmark and masquerade marked packets

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: newStatus := calculateStatus(rs, filteredPods, manageReplicasErr) // Always updates status as pods come up or die. <del> if err := updateReplicaSetStatus(rsc.kubeClient.Extensions().ReplicaSets(rs.Namespace), rs, newStatus); err != nil { </del> <ins> updatedRS, err := updateReplicaSetStatus(rsc.kubeClient.Extensions().ReplicaSets(rs.Namespace), rs, newStatus) if err != nil { </ins> // Multiple things could lead to this update failing. Requeuing the replica set ensures // Returning an error causes a requeue without forcing a hotloop return err } <ins> // Resync the ReplicaSet after MinReadySeconds as a last line of defense to guard against clock-skew. if manageReplicasErr == nil && updatedRS.Spec.MinReadySeconds > 0 && updatedRS.Status.ReadyReplicas == *(updatedRS.Spec.Replicas) && updatedRS.Status.AvailableReplicas != *(updatedRS.Spec.Replicas) { rsc.enqueueReplicaSetAfter(updatedRS, time.Duration(updatedRS.Spec.MinReadySeconds)*time.Second) } </ins> return manageReplicasErr }

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: } options := &generic.StoreOptions{RESTOptions: optsGetter, AttrFunc: GetAttrs} if err := store.CompleteWithOptions(options); err != nil { <del> panic(err) // TODO: Propagate error up </del> <ins> return nil, err </ins> } <del> return &REST{store} </del> <ins> return &REST{store}, nil </ins> } // Implement ShortNamesProvider

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When joining a new control plane node with stacked , while the local server starts and joins the cluster, can't connect to it and panics: What you expected to happen: Not sure if this is a bug per se, because in the end the static pod restarts and starts successfully after the server joins the cluster, but that panic should probably be gracefully handled. How to reproduce it (as minimally and precisely as possible): Join a new control plane node. Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: Hetzner VPS OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 5.3.0-45-generic (Ubuntu's default HWE kernel) Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
Yeah, the storage signature needs adjusting to propagate the error: /remove-kind bug /kind cleanup /priority backlog
/assign


Question: t.Errorf("Unexpected to find embedded.metadata.unknown=%#v", unknown) } <ins> t.Logf("Checking that metadata.generation=1") if generation, found, err := unstructured.NestedInt64(obj.UnstructuredContent(), "metadata", "generation"); err != nil { t.Errorf("unexpected error getting metadata.generation: %v", err) } else if !found { t.Errorf("expected metadata.generation=1: got: %d", generation) } else if generation != 1 { t.Errorf("unexpected metadata.generation=%d: expected this to be set to 1", generation) } </ins> t.Logf("Checking that ObjectMeta is pruned from wrongly-typed annotations") if annotations, found, err := unstructured.NestedStringMap(obj.UnstructuredContent(), "metadata", "annotations"); err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: func TestExtractFromDir(t *testing.T) { manifests := []api.ContainerManifest{ <del> {ID: "", Containers: []api.Container{{Image: "foo"}}}, {ID: "", Containers: []api.Container{{Image: "bar"}}}, </del> <ins> {Version: "v1beta1", Containers: []api.Container{{Name: "1", Image: "foo"}}}, {Version: "v1beta1", Containers: []api.Container{{Name: "2", Image: "bar"}}}, </ins> } files := make([]*os.File, len(manifests))

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: // makeHostsMount makes the mountpoint for the hosts file that the containers // in a pod are injected with. <del> func makeHostsMount(podDir, podIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) (*kubecontainer.Mount, error) { </del> <ins> func makeHostsMount(podDir, podIP, hostName, hostDomainName string, hostAliases []v1.HostAlias, useHostNetwork bool) (*kubecontainer.Mount, error) { </ins> hostsFilePath := path.Join(podDir, "etc-hosts") <del> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName, hostAliases); err != nil { </del> <ins> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName, hostAliases, useHostNetwork); err != nil { </ins> return nil, err } return &kubecontainer.Mount{

Currently, kubelet mounts hosts file for pods. For hostNetwork pods, the behavior is different and instead, kubelet relies on docker (or whatever container runtime) to mount the host's hosts file. We should have kubelet handle hosts file for hostNetwork pods. This would have several benefits: control over hosts file in case the container runtime does something consistent behavior between hostNetwork and non-hostNetwork pods consistent behavior between different container runtimes -managed hosts file enables features like to be used for hostNetwork pods Implementation would require: kubelet's or its caller output different content based on hostNetwork. i. If it's not hostNetwork, use existing behavior. ii. If it's using hostNetwork, read the content from the host's hosts file. hostNetwork constraint in deciding whether to mount the file Open Questions: do we treat changes to the host's hosts file? should the file's content be static between container restarts of the same Pod? or should every container restart pick up the latest content of the hosts file? the former seems better for consistency, but letter might be more in line with other features that use host resources like host volumes. condition to manage hosts file also excludes windows even though windows also the use of hosts file and the same format. seems like this should be supported also. windows-support ships, should it have validation against windows platform? the tricky bit is the client, API, and minions' kubelet may be running on different platforms, and currently, there's no precedence to platform validation. /kind feature /sig node


Question: collectors.NewVolumeStatsCollector(kl), collectors.NewLogMetricsCollector(kl.StatsProvider.ListPodStats), ) <ins> metrics.SetNodeName(kl.nodeName) </ins> // Setup filesystem directories. if err := kl.setupDataDirs(); err != nil {

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: curPod := cur.(*v1.Pod) oldPod := old.(*v1.Pod) if curPod.ResourceVersion == oldPod.ResourceVersion { <del> // Periodic resync will send update events for all known pods. </del> <ins> // In the event of a re-list we may receive update events for all known pods. </ins> // Two different versions of the same pod will always have different RVs. return }

<!-- Please only use this template for submitting enhancement requests --/area controller-manager What would you like to be : Don't resync statefulset in statefulset controller. Why is this needed: There are 2000+ sts in my k8s cluster, thus when i create a new sts, it takes a lot of time to create it's pod, i found that every sts will cost 40ms to sync it, then 2000 sts will cost 80s, and there is also a statefulSetResyncPeriod to resync all sts, I don't know why there need to have statefulSetResyncPeriod to resync all sts? Can we not resync sts and just watch it's changes?
/sig api-machinery
/remove-sig api-machinery /sig apps /sig scalability
the resync pattern is followed by all controllers to account for bugs in watch mechanism or missed notification due to any other reason. I dont think the individual controllers resync period is configurable or we could increase it . Also i am not sure if the current scale limits of 2000 StatefulSets is supported FYI
I have three cluster, A has 20 sts, B has 1300 sts and C has 2300 sts. I set kube-controller-manager loglevel to 4 and find that the cost of processing every sts is 2ms,30ms,45ms. Can we record the processed object into a processed cache and every time we get a new object to precess, we can compare it with the item in the cache, if they are same, then we don't need to sync it.
The "resync pattern" doesn't help with watch bugs (resync is done purely from internal cache, so in hypothetical situation that we missed watch event, it won't help at all). Resync-period should basically be used only in controllers that also sync some external things (like load-balancers), and watching changes of those is impossible. Unless I'm not missing anything, resync period can be safely dropped in this case.
I think all the controllers have the same problem because of the sharedinformerfactory has a default resync period if we don't set it to 0 in the config file. Also if there are more sts, the consumption for every sts will be larger.
interesting, but the resync is only happening for k8s objects.

Why is watching changes on those impossible ? And when you say external things you mean loadbalancers in gcloud ? resync is only happening for k8s services corresponding to those loadbalancers , right ? Or may be you are saying that if a k8s services is linked to a gcloud loadbalancer, the resync enables the controller code to go and check gcloud if the loadbalancing is provisioned or has anything changed there ? We need to document this in somwhere much like the apiconventions doc
- I mean loadbalancer in GCP as an example (but can also be other load balancer, persistent volumes in cloud or things like that). Yes - we resync on k8s objects, but that that let's you sync those. As an example - imagine service of type LoadBalancer - if you're periodically syncing the k8s service object, this triggers the operation of syncing the load-balancer (and we have an ability to check if it exists, is healthy etc.). - hmm, I would need to double check it, but I thought the default resync period is pretty long...
The default resync period is 12~24 hours. It is long but the bad experience will still happen every day.