Question: utilstrings "k8s.io/utils/strings" ) <ins> func getPath(uid types.UID, volName string, host volume.VolumeHost) string { return host.GetPodVolumeDir(uid, utilstrings.EscapeQualifiedName(nfsPluginName), volName) } </ins> // ProbeVolumePlugins is the primary entrypoint for volume plugins. <ins> // This is the primary entrypoint for volume plugins. </ins> // The volumeConfig arg provides the ability to configure recycler behavior.  It is implemented as a pointer to allow nils. // The nfsPlugin is used to store the volumeConfig and give it, when needed, to the func that creates NFS Recyclers. // Tests that exercise recycling should not use this func but instead use ProbeRecyclablePlugins() to override default behavior.

/kind feature What happened: on a cluster with an nfs persistent volume present: What you expected to happen: I expected metrics concerning the nfs persistent volume to be returned e.g: (from a separate aws cluster) Anything else we need to know?: I'm assuming that this is just not implemented currently? I couldn't find a referencing issue Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on-prem OS (e.g. from /etc/os-release):Kernel (e.g. ):
/sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/open Metrics is missing for NFS driver.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/sig storage


Question: } } <ins> client := NewDiscoveryClientForConfigOrDie(&restclient.Config{Host: server.URL}) start := time.Now() </ins> serverResources, err := client.ServerResources() if err != nil { t.Errorf("unexpected error: %v", err) } <ins> end := time.Now() if d := end.Sub(start); d > time.Second { t.Errorf("took too long to perform discovery: %s", d) } </ins> serverGroupVersions := groupVersions(serverResources) <del> expectedGroupVersions := []string{"v1", "extensions/v1beta1", "extensions/v1beta2"} </del> <ins> expectedGroupVersions := []string{ "v1", "apps/v1beta1", "apps/v1beta2", "apps/v1beta3", "apps/v1beta4", "apps/v1beta5", "apps/v1beta6", "apps/v1beta7", "apps/v1beta8", "apps/v1beta9", "apps/v1beta10", "extensions/v1beta1", "extensions/v1beta2", "extensions/v1beta3", "extensions/v1beta4", "extensions/v1beta5", "extensions/v1beta6", "extensions/v1beta7", "extensions/v1beta8", "extensions/v1beta9", "extensions/v1beta10", } </ins> if !reflect.DeepEqual(expectedGroupVersions, serverGroupVersions) { t.Errorf("unexpected group versions: %v", diff.ObjectReflectDiff(expectedGroupVersions, serverGroupVersions)) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: // TestBindPlugin tests invocation of bind plugins. func TestBindPlugin(t *testing.T) { testContext := testutils.InitTestAPIServer(t, "bind-plugin", nil) <del> bindPlugin1 := &BindPlugin{PluginName: "bind-plugin-1", client: testContext.ClientSet} bindPlugin2 := &BindPlugin{PluginName: "bind-plugin-2", client: testContext.ClientSet} </del> <ins> bindPlugin1 := &BindPlugin{name: "bind-plugin-1", client: testContext.ClientSet} bindPlugin2 := &BindPlugin{name: "bind-plugin-2", client: testContext.ClientSet} </ins> reservePlugin := &ReservePlugin{name: "mock-reserve-plugin"} postBindPlugin := &PostBindPlugin{name: "mock-post-bind-plugin"} // Create a plugin registry for testing. Register reserve, bind, and

See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: "strconv" "sync" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/client/cache" "k8s.io/kubernetes/pkg/runtime"

We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.


Question: rbacv1helpers.NewRule("update").Groups(appsGroup).Resources("statefulsets/finalizers").RuleOrDie(), rbacv1helpers.NewRule("get", "create", "delete", "update", "patch").Groups(legacyGroup).Resources("pods").RuleOrDie(), rbacv1helpers.NewRule("get", "create", "delete", "update", "patch", "list", "watch").Groups(appsGroup).Resources("controllerrevisions").RuleOrDie(), <del> rbacv1helpers.NewRule("get", "create").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie(), </del> <ins> rbacv1helpers.NewRule("get", "create", "list", "watch").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie(), </ins> eventsRule(), }, } if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetAutoDeletePVC) { role.Rules = append(role.Rules, rbacv1helpers.NewRule("update", "delete").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie()) <ins> role.Rules = append(role.Rules, rbacv1helpers.NewRule("update").Groups(legacyGroup).Resources("pods/finalizers").RuleOrDie()) </ins> } return role

When OwnerReferencesPermissionEnforcement admission plugin is used the PVCs are not deleted and KCM logs show PVCs should have ownerReferences set and the PVCs should be garbage collected on scale down 3 of 6 of the following tests fail when OwnerReferencesPermissionEnforcement admission plugin is turned on No response <details</details<detailsNA </details<details</details<details</details<details</details<details</details
/sig apps /triage accepted


Question: extension = "tar.gz", package_dir = "kubernetes", strip_prefix = "//", <ins> symlinks = { "kubernetes/cluster/gce/cos/": "gci", "kubernetes/cluster/gce/custom/": "gci", "kubernetes/cluster/gce/ubuntu/": "gci", }, </ins> tags = ["no-cache"], deps = [ ":_full_server",

What happened: that uses Ubuntu image fails during Kubernetes installation: Full log: Indeed, is not present on nodes. Only and subdirectories are present in . I don't know what code removed (or did not package) and symlinks, but they should be there. What you expected to happen: Kubernetes is installed correctly and e2e tests start. How to reproduce it (as minimally and precisely as possible): Run in a PR. Anything else we need to know?: Prow job: It seems to be the only one that uses Ubuntu image. This command works and installs Kubernetes correctly: is present there. Dunno what the prow job does differently.
/sig testing
the nodes receive files staged via the release scripts and downloaded by the cluster/ scripts, I don't think sig testing will be the right group to fix this. SIG node has some owners related to the various images that might have some idea how this is setup
/sig release this is probably an issue in the release scripts
cc:
PTAL, it looks like a stupid bug somewhere.
Q: Did this ever run successfully?
Most probably not, I haven't found a working e2e tests on Ubuntu. Still, works well, there must be something slightly different in test setup scripts.
ping again? Short recap: This works: is there. ( is symlink to ) This does not work: Fails with: does not exist. The prow job must do something slightly different than kube-
The serial job is using GCI (COS): Is the non-serial job working? It uses ubuntu: If not, my guess from the last time I remember looking was checking the release bundle and ensuring it includes the ubuntu dir
No, same error. created by contains , released bundles (e.g. ) contains it too. Otherwise would not work with an Ubuntu image. So what does the prow job differently? Does something there prune the scripts? is just a symlink to , does it make a difference?
Found it, creates without any symlinks. It looks like bazel limitation. I filled to explicitly create the symlinks when tarball is created. BTW, if and use different way to produce tarballs, who / when tests the ones created by ?
Most of CI is using make release or equivalent. Presumbit uses bazel. Jan Šafránek <:
/area release-eng /milestone v1.16 /priority important
/priority important-soon


Question: for { conn, err := listener.AcceptSCTP() assertNoError(err, fmt.Sprintf("failed accepting SCTP connections")) <del> clientAddress := conn.RemoteAddr().String() </del> <ins> remoteAddr, err := conn.SCTPRemoteAddr(0) if err != nil { assertNoError(err, "failed to get SCTP client remote address") } clientAddress := remoteAddr.String() </ins> n, err := conn.Read(buf) assertNoError(err, fmt.Sprintf("failed to read from SCTP client %s", clientAddress)) receivedText := strings.ToLower(strings.TrimSpace(string(buf[0:n])))

How to repro: an agnhost pod listening on sctp into the pod and use the connect command to probe against the same pod The stack trace that points to and this is from the vendored library , /kind bug /sig network /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Run an agnhost pod listening on sctpExec into the pod and use the connect command to probe against the same pod The stack trace that points to /sig network /help </details
/assign
/triage accepted
here we are immediately closing the connection after successful connect leading to EOF at server end (it is expecting data) causing the failure. We can tweak this to connect send some data just like UDP and then exit? Works locally writing "hostname" to the connection. [EDIT] For me it sometimes failed with EOF, sometime with nil pointer exception at . This reads sock opts via syscall and we immediately close the connection .. kind of a race between the client closing connection and server reading fds?
the panic seems a bug in that project, no? Also SCTP, has a establishment handshake, so the data should not be relevant in this case
returns fd and socket address, for is used to implement (always read from struct field). The SCTP implementation discards the returned socket address by call. , and is always a new syscall (). In our case the client immediately closes the connection, there can be chances where SCTPGETPEERADDRS will fail and will return nil in these cases. I think this is reasonable for their project. We call and panic is raised by calling on nil object. We can easily avoid this by not null check.

Also, I strongly believe that the client handling should be done in a separate go-routine instead of the main loop to avoid errors like this crashing the pod. [EDIT] The client in our case needs to send some data after connection, or else the server will throw EOF and we currently have assertions that use for errors.
panicing does not seem very reasonable It looks to me that the problem is that the SCTPConn struct needs to check if the connection is still valid before trying to access the file descriptor , see the std library implementation and then, all methods implement this check to me it is a bug on the implementation
The implementation is not causing any panic, it simply returns nil. Panic is caused when we try to call on nil object.
oh, sorry, my bad I should read your comment with more detail, is well explained there I recognize I have zero idea of how sctp works internally, maybe we can just be more resilient on our tests and code and detect if is nil and avoid getting into the complexity of this protocol ... sending some random data also sounds good, we do this for udp IIRC
are you able to send a PR to add those improvements to the test image?


Question: ) go_genrule( name = "zz_generated.openapi", <del> srcs = srcs + ["//hack/boilerplate:boilerplate.go.txt"], </del> <ins> srcs = srcs + ["//" + vendor_prefix + "hack/boilerplate:boilerplate.go.txt"], </ins> outs = ["zz_generated.openapi.go"], cmd = " ".join([ "$(location //vendor/k8s.io/code-generator/cmd/openapi-gen)", "--v 1", "--logtostderr", <del> "--go-header-file $(location //hack/boilerplate:boilerplate.go.txt)", </del> <ins> "--go-header-file $(location //" + vendor_prefix + "hack/boilerplate:boilerplate.go.txt)", </ins> "--output-file-base zz_generated.openapi", <del> "--output-package k8s.io/kubernetes/pkg/generated/openapi", "--input-dirs " + ",".join(["k8s.io/kubernetes/" + target for target in openapi_targets] + ["k8s.io/kubernetes/vendor/" + target for target in vendor_targets]), "&& cp pkg/generated/openapi/zz_generated.openapi.go $(location :zz_generated.openapi.go)", </del> <ins> "--output-package " + go_prefix + vendor_prefix + "pkg/generated/openapi", "--input-dirs " + ",".join([go_prefix + target for target in openapi_targets] + [go_prefix + "vendor/" + target for target in vendor_targets]), "&& cp " + vendor_prefix + "pkg/generated/openapi/zz_generated.openapi.go $(location :zz_generated.openapi.go)", </ins> ]), go_deps = deps, tools = ["//vendor/k8s.io/code-generator/cmd/openapi-gen"],

As part of moving federation out of tree (), the federation build will need to generate ` for itself and vendored kubernetes. The following steps are required: [x] Discover targets for generation () [x] Update the file's bazel rules to configurably enable a build for vendored kube () [x] Support generation of the file to a vendored path () [x] Update kubernetes to use latest kazel and kube-openapi () /sig multicluster testing /area federation /assign cc:
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: import ( "context" "testing" <ins> "time" </ins> "k8s.io/klog/v2"

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: // Queried from cloud provider, if available. Addresses []NodeAddress `json:"addresses,omitempty" description:"list of addresses reachable to the node" patchStrategy:"merge" patchMergeKey:"type"` // NodeSystemInfo is a set of ids/uuids to uniquely identify the node <del> NodeInfo NodeSystemInfo `json:"nodeInfo,omitempty"` </del> <ins> NodeInfo NodeSystemInfo `json:"nodeInfo,omitempty" description:"set of ids/uuids to uniquely identify the node"` </ins> } type NodePhase string

PR had a few errors in its API objects, where the "description" field errors in its JSON declarations were missing. But, the "verify-" presubmit check didn't flag them. This should be investigated and fixed.
Could you please take a look at this?


Question: Name:      "node2mirrorpod", Namespace: "ns", }, <ins> DeleteOptions: &metav1.DeleteOptions{GracePeriodSeconds: &zero}, </ins> }) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set.

Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


Question: {"1234/5678": "bar"}, {"1.2.3.4/5678": "bar"}, {"UpperCase123": "bar"}, <del> {"a": strings.Repeat("b", totalAnnotationSizeLimitB-1)}, </del> <ins> {"a": strings.Repeat("b", TotalAnnotationSizeLimitB-1)}, </ins> { <del> "a": strings.Repeat("b", totalAnnotationSizeLimitB/2-1), "c": strings.Repeat("d", totalAnnotationSizeLimitB/2-1), </del> <ins> "a": strings.Repeat("b", TotalAnnotationSizeLimitB/2-1), "c": strings.Repeat("d", TotalAnnotationSizeLimitB/2-1), </ins> }, } for i := range successCases {

Expose totalAnnotationSizeLimitB in the validation package: This value is useful for validating annotations outside of the validation package, such as in SSA: Otherwise, size validation on annotations outside of the package would diverge and cause issues later. /kind cleanup /wg api-expression cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: framework.NewFrameworkExtensions = append(framework.NewFrameworkExtensions, func(f *framework.Framework) { ginkgo.AfterEach(func() { <ins> if f.ClientSet == nil { // Test didn't reach f.BeforeEach, most // likely because the test got // skipped. Nothing to check... return } </ins> e2enode.AllNodesReady(f.ClientSet, 3*time.Minute) }) },

After merged, build logs for e2e tests have started showing stack traces that look like: It seems like the used in this is nil I have a suspicion this is happening for storage tests that skip No new stack traces in build logs Run e2e tests against the in-development version of kubernetes I identified when this started happening by looking for an increase in build-log size is the first build where this started happening, which happens to be the first build after merged, which is the PR that introduced this new way of invoking <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing /triage accepted /assign To put this on your radar
/cc
Correct. This happens because of the ordering in Tests gets skipped before is called. This avoids the expensive namespace creation/deletion for tests that don't need to run. The for then gets executed although the framework didn't get initialized. My preferred solution is to document that the framework instance might not be initialized in an AfterEach callback and add a nil check to .
See Only was affected. The other framework extensions use DeferCleanup, which only runs when needed.


Question: // propagateControllers on an unified hierarchy enables all the supported controllers for the specified cgroup func propagateControllers(path string) error { <del> if err := os.MkdirAll(path, 0755); err != nil { </del> <ins> if err := os.MkdirAll(filepath.Join(cmutil.CgroupRoot, path), 0755); err != nil { </ins> return fmt.Errorf("failed to create cgroup %q : %v", path, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is created on a cgroup2 host What you expected to happen: Such directory should not be created How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 with Install moby/moby with containerd/containerd and opencontainers/runc Install kubernetes-sigs/kind . This version uses as the base image. Run with Run Run to drop into the shell of the node The following directories are unexpectedly created. (No process seems to belong to these groups actually) Environment: (See above)
cc
/sig node


Question: return fmt.Errorf("failed to listen on %q: %v", s.endpoint, err) } // Create the grpc server and register runtime and image services. <del> s.server = grpc.NewServer() </del> <ins> s.server = grpc.NewServer( grpc.MaxRecvMsgSize(maxMsgSize), grpc.MaxSendMsgSize(maxMsgSize), ) </ins> runtimeapi.RegisterRuntimeServiceServer(s.server, s.service) runtimeapi.RegisterImageServiceServer(s.server, s.service) go func() {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Nodes in status Pods in status What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian 8 (jessie) Kernel (e.g. ): Install tools: kops Others: Logs:Possible prerequisites: a lot of jobs (1657) that failed (state: Error) because of internal problems Cluster size: 3 masters, 4 worker nodes Workaround: how to get the node backPossible related to
/sig node
It is limitation on grpc. The default value is (i.e. in error message)
do you know if it possible to configure the value with environment variable or maybe in some config file?
Hitting the same bug here (Kubernetes 1.10.2 with docker 17.12.1). The node has 8309 containers, most of them exited, created by failing jobs.
I'll try to change kubelet's ContainerGC so that, when there are more than say 5000 containers it reclaims the oldest ones. Unless someone thinks it's not the right think to do.
can you please try ?
thanks I'll try to reproduce but it will take time (I had to clean the host).
I don't have free infrastructure to install there K8s from some branch and don't have experience to do this at all. I have used only stable releases.
I can already tell that it probably won't solve the problem (with containerd at least):
given what you are seeing won't help but will help. right? (note that default in grpc is )
yes it would work but it's 1GB so we are in a runaway situation here; have 1GB today and you'll need 2GB tomorrow. 16MB is definitely an improvement and your change to 16MB is LGTM (thanks!), but the root problem still remains. The 2 solutions I can imagine are (1) an API change to include paging (or streaming), or (2) make the kubelet ensure there's less than a total number of containers (running+exited).

Since the issue also impacts I'm participating in the following upstream issue:
Just discovered the way of the stream seems to be the target:
improves the situation but doesn't completely fix this issue ;-) Should I open another one?
k8s-merge-robot is a bot user; I think you need to open a new issue
it was a humoristic remark ;-) I'm waiting for the improvement in containerd. It seems they're even gonna hide the details about List vs Stream in the client, so we may just have to bump de client version at some point here.
Exactly! We just downed a docker swarm mode cluster with mere 80 services with 1 container each.
run into this issue as well, dead container did not remove by gc, this functionality seems deprecated in recent k8s release, no sure why. see
I met the problem for k8s 1.13.2 .Do you have any solutions except clear dead container manually?
Really? There's an absolute universe more information in 1GB that 4MB,, just as it made sense to not make the limit 4096. Although the above is true, can we agree there is huge difference, in human terms, between 4MB and say 1GB. Pretty please change the limit to a round 1GB or something that is configurable at run-time by the service publisher or as a connection option. Maybe an option flag that waves this limit at runtime? This is a scary issue for us b/c now we have to write a segmenter/desegmenter in our client? Fear. Please let consumers of pb tech not have to do this. Everyone will just have to write their own segmenter/desegmenter hurting the pb's rep as the one stop shop for interoperabilty.
streaming is native in protobuf...
Sure... but we're hitting this limit and we're already in a stream call. :
On Jul 21, 2019 17:36, drew-512 <: Sure... but we're hitting this limit and we're already in a stream call. :Sounds weird, you're talking about the limit of the Kubernetes' kubelet gRPC client talking to the CRI runtime? We never got this bug again and it's an old one.

While doing some performance testing with AWS EKS, I ran into a situation where pods would have "CreateContainerError". These were created in a non-default namespace on CentOS 7.4 hosts. Within this cluster - remember it is used for performance testing - there were over 20,000 deployments, all in a failed state. Worker nodes would have entries like: These pods were in a namespace other than default. However, I was able to successfully scale coredns as desired without a problem. So I learned this problem was limited to a namespace. I deleted the namespace and recreated it and I could successfully deploy pods. docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?
On Jul 22, 2019 18:31, Dave Augustus <:docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?AFAIK using docker does not go through the same interface as containerd. You should probably be able to recover the situation by removing containers (ie: docker system prune).
For anyone operating a k8s cluster, running into this error and not being sure what the actual problem is... If you run this , you'll see that a message describing all containers currently known to docker is bigger than the limit the Kubelet has set for gRPC messages (16MB at the time of writing). This either means you have a LOT of (possibly dead) containers, or there's a LOT of metadata attached to your containers. This metadata probably comes in the form of custom labels attached to the container image by whatever tool you're using to create the container images. Check out for a couple of your containers and the issue should become apparent.
Thanks your comment was really helpful!! However, is there any systematic solution to resolve this? rather than on such node or remove dead containers manually? I mean, is there any chance to increase limit of gRPC message size?
My case was caused by too many tekton dead pod (over 3000 dead containers).

I will try to add to the kubelet to solve the issue
Thanks for sharing! :+1:
My case was caused by too many tekton dead pod (over 5000 dead containers). the kubelet get the message to solve this issue. first , add to kubelet comand line . second . restart kubelet third. recheck and problem is solved. thanks to
I'm glad it helped. In my case, after adding , it still happened once after. So I have to add a docker-prune service for double insurance. Just for reference /etc/systemd/system/docker-prune.service /etc/systemd/system/docker- It will run once a week on Sunday Using systemctl enable and start both. check the status with .
The solution here for me, install nerdctl, then: Once containers are pruned, you need to restart containerd for the node to show ready again.


Question: framework.Failf("Error cleanup pod `%s/%s`: %v", metav1.NamespaceSystem, "critical-pod", err) } }() <del> runPausePodWithTimeout(ctx, f, pausePodConfig{ </del> <ins> runPausePod(ctx, f, pausePodConfig{ </ins> Name:              "critical-pod", Namespace:         metav1.NamespaceSystem, PriorityClassName: scheduling.SystemClusterCritical,

master-blocking kind-ipv6-master-parallel RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taints AdmissionWebhook [Privileged:ClusterAdmin] should honor timeout [Conformance] 01-27-2023 16:08 PST https://k8s- { failed [FAILED] timed out while waiting for pod runtimeclass-3517/without-label to be running In [It] at: @ 01/28/23 22:33:45.607 } No response kind cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It looks like that “Kubernetes e2e suite: . Would we want to reopen this issue? I will keep a close eye on to see if it continues to flake.
Reopen to monitor failures cc
/sig node /cc
/retitle [It] [sig-node] RuntimeClass should run a Pod requesting a RuntimeClass with scheduling without taint
Example failure:
Failure The pod was created at , from the kubelet logs: so the pod started at , but it was deleted at because the 1 minute timeout was hit. It looks like the container startup took a while based on containerd logs: there is a gap there of and between and
It is hitting this timeout - That uses which is 1 min which seems to be too short. We should probably use or which is 5min or 2min, because the whole pod needs to start and it seems like 1min is too short in this case.


Question: srcs = [ ":package-srcs", "//hack/boilerplate:all-srcs", <ins> "//hack/conformance:all-srcs", </ins> "//hack/e2e-internal:all-srcs", "//hack/lib:all-srcs", "//hack/make-rules:all-srcs",

There are a number of conformance test requirements and guidelines: We should try to automate enforcement to the degree we can, via a "linting" tool, and/or test framework checks. For instance, conformance tests shouldn't invoke framework.ProviderIs or framework.SkipUnlessProviderIs. Those could actually check whether they are invoked by a conformance test, and then die. cc /area conformance
/sig testing
/cc


Question: selectPolicyFn = max // Use the default policy otherwise to produce a highest possible change } for _, policy := range scalingRules.Policies { <del> replicasAddedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleEvents) periodStartReplicas := currentReplicas - replicasAddedInCurrentPeriod </del> <ins> replicasAddedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleUpEvents) replicasDeletedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleDownEvents) periodStartReplicas := currentReplicas - replicasAddedInCurrentPeriod + replicasDeletedInCurrentPeriod </ins> if policy.Type == autoscalingv2.PodsScalingPolicy { proposed = periodStartReplicas + policy.Value } else if policy.Type == autoscalingv2.PercentScalingPolicy {

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // adds a watcher to the bucket, if the deadline is before the start, it will be // added to the first one. func (t *watcherBookmarkTimeBuckets) addWatcher(w *cacheWatcher) bool { <del> nextTime, ok := w.nextBookmarkTime(t.clock.Now()) if !ok { return false } </del> <ins> nextTime := w.nextBookmarkTime(t.clock.Now()) </ins> bucketID := nextTime.Unix() t.lock.Lock() defer t.lock.Unlock()

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: name = "all-srcs", srcs = [ ":package-srcs", <ins> "//pkg/kubelet/server/metrics:all-srcs", </ins> "//pkg/kubelet/server/portforward:all-srcs", "//pkg/kubelet/server/remotecommand:all-srcs", "//pkg/kubelet/server/stats:all-srcs",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: NodeCIDRMaskSizes:    []int{24, 98, 24}, }, }, <ins> { description: "no double counting", fakeNodeHandler: &testutil.FakeNodeHandler{ Existing: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{ Name: "node0", }, Spec: v1.NodeSpec{ PodCIDRs: []string{"10.10.0.0/24"}, }, }, { ObjectMeta: metav1.ObjectMeta{ Name: "node1", }, Spec: v1.NodeSpec{ PodCIDRs: []string{"10.10.2.0/24"}, }, }, { ObjectMeta: metav1.ObjectMeta{ Name: "node2", }, }, }, Clientset: fake.NewSimpleClientset(), }, allocatorParams: CIDRAllocatorParams{ ClusterCIDRs: func() []*net.IPNet { _, clusterCIDR, _ := net.ParseCIDR("10.10.0.0/22") return []*net.IPNet{clusterCIDR} }(), ServiceCIDR:          nil, SecondaryServiceCIDR: nil, NodeCIDRMaskSizes:    []int{24}, }, expectedAllocatedCIDR: map[int]string{ 0: "10.10.1.0/24", }, }, </ins> } // test function testFunc := func(tc testCase) { <ins> fakeNodeInformer := getFakeNodeInformer(tc.fakeNodeHandler) nodeList, _ := tc.fakeNodeHandler.List(context.TODO(), metav1.ListOptions{}) </ins> // Initialize the range allocator. <del> allocator, err := NewCIDRRangeAllocator(tc.fakeNodeHandler, getFakeNodeInformer(tc.fakeNodeHandler), tc.allocatorParams, nil) </del> <ins> allocator, err := NewCIDRRangeAllocator(tc.fakeNodeHandler, fakeNodeInformer, tc.allocatorParams, nodeList) </ins> if err != nil { t.Errorf("%v: failed to create CIDRRangeAllocator with error %v", tc.description, err) return

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With a /22 pod network and a /24 assigned to each node, a third node cannot get a pod CIDR if kube-controller-manager has ever been restarted. What you expected to happen: Once a third or fourth node joins, it gets a /24. How to reproduce it (as minimally and precisely as possible): 150% usage on the IPAM controller is not a good sign. Anything else we need to know?: During the creation of the rangeAllocator, Occupy() is called twice on the pod networks. The first time is before subscribing to the node informer. The second time is during the node informer callbacks because on subscribing it calls Add for all existing nodes. I will submit a proposed fix shortly. Environment: Kubernetes version (use ): 1.19.2
/sig network
/triage accepted thanks for finding the bug I 've introduced :)


Question: - `alpha` support (guarded by the `ServiceAccountTokenJTI` feature gate) for adding a `jti` (JWT ID) claim to service account tokens it issues, adding an `authentication.kubernetes.io/credential-id` audit annotation in audit logs when the tokens are issued, and `authentication.kubernetes.io/credential-id` entry in the extra user info when the token is used to authenticate. - `alpha` support (guarded by the `ServiceAccountTokenPodNodeInfo` feature gate) for including the node name (and uid, if the node exists) as additional claims in service account tokens it issues which are bound to pods, and `authentication.kubernetes.io/node-name` and `authentication.kubernetes.io/node-uid` extra user info when the token is used to authenticate. - `alpha` support (guarded by the `ServiceAccountTokenNodeBinding` feature gate) for allowing `TokenRequests` that bind tokens directly to nodes, and (guarded by the ServiceAccountTokenNodeBindingValidation feature gate) for validating the node name and uid still exist when the token is used. ([#120780](https://github.com/kubernetes/kubernetes/pull/120780), [@munnerz](https://github.com/munnerz)) <ins> - `kube-apiserver` updated: - Updated encryption configuration file watch logic from using inotify watch to polling at an interval of every minute. Moved the logic to polling because there are variations on file changes (like symlink swapping of directories that contain the encryption config) that the file watch logic would fail to detect. Polling at a set interval prevents any such issues. Note that there is no guarantee on how quickly the API server will process the encryption config. The `apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds` metric must be used to determine when the new config becomes effective.([#121310](https://github.com/kubernetes/kubernetes/pull/121310), [@nilekhc](https://github.com/nilekhc)) </ins> - `kube-controller-manager`: The `LegacyServiceAccountTokenCleanUp` feature gate is now `beta` and enabled by default. When enabled, legacy auto-generated service account token secrets are auto-labeled with a `kubernetes.io/legacy-token-invalid-since` label if the credentials have not been used in the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year), **and** are referenced from the `.secrets` list of a ServiceAccount object, **and**  are not referenced from pods. This label causes the authentication layer to reject use of the credentials. After being labeled as invalid, if the time specified by `--legacy-service-account-token-clean-up-period` (defaulting to one year) passes without the credential being used, the secret is automatically deleted. Secrets labeled as invalid which have not been auto-deleted yet can be re-activated by removing the `kubernetes.io/legacy-token-invalid-since` label. ([#120682](https://github.com/kubernetes/kubernetes/pull/120682), [@yt2985](https://github.com/yt2985)) - `kube-proxy` will only install the `DROP` rules for invalid `conntrack` states if the `nf_conntrack_tcp_be_liberal` is not set. ([#120412](https://github.com/kubernetes/kubernetes/pull/120412), [@aojea](https://github.com/aojea))

updated the automatic reloading logic from file watch to polling. This change in behavior needs to be documented [ ] Update the v1.29 release notes to call out this change in behavior (ref: ) to replace references to file watch with polling. /sig auth /triage accepted /assign nilekhc


Question: if o.taintsToAdd, o.taintsToRemove, err = parseTaints(taintArgs); err != nil { return cmdutil.UsageError(cmd, err.Error()) } mapper, typer := f.Object() o.builder = resource.NewBuilder(mapper, f.CategoryExpander(), typer, resource.ClientMapperFunc(f.ClientForMapping), f.Decoder(true)). ContinueOnError(). NamespaceParam(namespace).DefaultNamespace() <ins> if o.selector != "" { o.builder = o.builder.SelectorParam(o.selector).ResourceTypes("node") } </ins> if o.all { <del> o.builder = o.builder.SelectAllParam(o.all).ResourceTypes("node") } else { if len(o.resources) < 2 { return fmt.Errorf("at least one resource name must be specified since 'all' parameter is not set") } </del> <ins> o.builder = o.builder.SelectAllParam(o.all).ResourceTypes("node").Flatten().Latest() } if !o.all && o.selector == "" && len(o.resources) >= 2 { </ins> o.builder = o.builder.ResourceNames("node", o.resources[1:]...) } o.builder = o.builder.SelectorParam(o.selector). Flatten(). Latest() o.f = f o.out = out o.cmd = cmd <ins> return nil } // validateFlags checks for the validation of flags for kubectl taints. func (o TaintOptions) validateFlags() error { // Cannot have a non-empty selector and all flag set. They are mutually exclusive. if o.all && o.selector != "" { return fmt.Errorf("setting 'all' parameter with a non empty selector is prohibited.") } // If both selector and all are not set. if !o.all && o.selector == "" { if len(o.resources) < 2 { return fmt.Errorf("at least one resource name must be specified since 'all' parameter is not set") } else { return nil } } </ins> return nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: dq := NewDelayQueue() delay := 2 * time.Second <del> dq.Add(&testjob{d: delay}) </del> before := time.Now() <ins> dq.Add(&testjob{d: delay}) </ins> x := dq.Pop() now := time.Now()

Failure:
Assigning to per
do you have reason to think this is still happening? if not, want to close it?
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Test passed with the latest code, suggest to close it firstly; if happened again, please assign it to me directly :). Head: OS: Test command: Test output:
- this is flake, not permanent failure (as almost all issues with "kind/flake"; running a test once most probably will not fail and that's expected
thanks for your input. The log is gone, let me check the code to see if any clue :).
Here maybe the root case after reviewing the code: the do the following steps: into current time () item from current time () the wait period () The problem is that: the timer start at "Step 1" in in , if there's any delay between "Step 1" & "Step 2", the wait period will be shorter than expected. The proposed PR is to record time before adding item. If any comments, please let me know. Thanks klaus


Question: return false } <ins> // hasNonLocalProjectedTokenPath return true if spec.Volumes have any entry with non-local projected token path func hasNonLocalProjectedTokenPath(spec *api.PodSpec) bool { for _, volume := range spec.Volumes { if volume.Projected != nil { for _, source := range volume.Projected.Sources { if source.ServiceAccountToken == nil { continue } errs := apivalidation.ValidateLocalNonReservedPath(source.ServiceAccountToken.Path, nil) if len(errs) != 0 { return true } } } } return false } </ins> // GetValidationOptionsFromPodSpecAndMeta returns validation options based on pod specs and metadata func GetValidationOptionsFromPodSpecAndMeta(podSpec, oldPodSpec *api.PodSpec, podMeta, oldPodMeta *metav1.ObjectMeta) apivalidation.PodValidationOptions { // default pod validation options based on feature gate

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: _ "k8s.io/code-generator/cmd/informer-gen" _ "k8s.io/code-generator/cmd/lister-gen" _ "k8s.io/code-generator/cmd/register-gen" <ins> _ "k8s.io/kube-openapi/cmd/openapi-gen" </ins> )

We're trying to upgrade to () and found the issue on executing (). The problem is that on () missed dependency that not allow to install () No error To reproduce the same what happening on , just execute this command on code-generator folder: I think it's happens after this commit <details</details<details</details<details</details<details</details<details</details<details</details
/cc
/assign
/sig api-machinery
I was able to reproduce, but just because I had an older version of kube-openapi in my GOPATH. Using the prefix in the command fixed it. I think line needs to be
Yeah, with works for me too.
Either that or we pin to a more recent commit, for example . For some reason if you ommit the version it is using a from Feb 27.
thanks for investigating. Question, if we use would this not cause problems in the future if we try to build a release branch from the past? Maybe this is not a big deal. OTOH if we hardcode a tag, it may create maintenance costs. I'm wondering if there is a better way, say to version and use the corresponding tag, say ?
Using tags would be a way to have more reproducible builds, they still need to be tagged from time to time tho. That is still better than using or micro managing commit revisions.
Thanks and for the research. Is either of you thinking on submitting a fix for this? Thanks!
I can do it.
Thanks! /assign mbobrovskyi
Yes, please don't use Generally instead we'd control the version of a tool in the hack/tools module and install it from there (see e.g.
This allows us to also gain for the package while not polluting the main module with dev-only dependencies.
Not sure if we need to make another Because it's not just a dev-only dependency. I propose to add it to of code-generator.
This is about semversioning
/triage accepted


Question: } function expand-default-subnetwork() { <del> gcloud compute networks switch-mode "${NETWORK}"  --mode custom  </del> <ins> gcloud compute networks update "${NETWORK}"  --switch-to-custom-subnet-mode  </ins> --project "${NETWORK_PROJECT}"  --quiet || true gcloud compute networks subnets expand-ip-range "${NETWORK}"

/kind bug What happened: Warning printed by : What you expected to happen: No warning. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This is my first e2e cluster. Logs with more context below. Environment: Kubernetes version (use ): master Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): Description: Ubuntu 14.04 LTS
This is probably an issue in the scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
This has started failing in tests: https://k8s-
/shrug /milestone v1.10 /assign /assign /priority failing-test /priority critical-urgent /sig network /sig testing
I think this might be only in the staging gcloud release; at least doesn't mention this being removed.
and yeah, gce-canary is also using staging gcloud /shrug
/assign I'll fix kubetest, is patching k8s
/approved-for-milestone
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/status in-review
should we close this issue since the PR merged?
waiting for , and that need to be cherrypicked into older releases
still need to cherrypick into previous releases
agreed, but isn't blocking 1.10 any more /milestone clear
cherrypick is done, closing /close


Question: // can reasonably expect seems questionable. {Group: "extensions", Version: "v1beta1"}: {group: 17900, version: 1}, // to my knowledge, nothing below here collides <del> {Group: "apps", Version: "v1"}:                              {group: 17800, version: 15}, {Group: "events.k8s.io", Version: "v1beta1"}:                {group: 17750, version: 5}, {Group: "authentication.k8s.io", Version: "v1"}:             {group: 17700, version: 15}, {Group: "authentication.k8s.io", Version: "v1beta1"}:        {group: 17700, version: 9}, {Group: "authorization.k8s.io", Version: "v1"}:              {group: 17600, version: 15}, {Group: "authorization.k8s.io", Version: "v1beta1"}:         {group: 17600, version: 9}, {Group: "autoscaling", Version: "v1"}:                       {group: 17500, version: 15}, {Group: "autoscaling", Version: "v2beta1"}:                  {group: 17500, version: 9}, {Group: "autoscaling", Version: "v2beta2"}:                  {group: 17500, version: 1}, {Group: "batch", Version: "v1"}:                             {group: 17400, version: 15}, {Group: "batch", Version: "v1beta1"}:                        {group: 17400, version: 9}, {Group: "batch", Version: "v2alpha1"}:                       {group: 17400, version: 9}, {Group: "certificates.k8s.io", Version: "v1beta1"}:          {group: 17300, version: 9}, {Group: "networking.k8s.io", Version: "v1"}:                 {group: 17200, version: 15}, {Group: "networking.k8s.io", Version: "v1beta1"}:            {group: 17200, version: 9}, {Group: "policy", Version: "v1beta1"}:                       {group: 17100, version: 9}, {Group: "rbac.authorization.k8s.io", Version: "v1"}:         {group: 17000, version: 15}, {Group: "rbac.authorization.k8s.io", Version: "v1beta1"}:    {group: 17000, version: 12}, {Group: "rbac.authorization.k8s.io", Version: "v1alpha1"}:   {group: 17000, version: 9}, {Group: "settings.k8s.io", Version: "v1alpha1"}:             {group: 16900, version: 9}, {Group: "storage.k8s.io", Version: "v1"}:                    {group: 16800, version: 15}, {Group: "storage.k8s.io", Version: "v1beta1"}:               {group: 16800, version: 9}, {Group: "storage.k8s.io", Version: "v1alpha1"}:              {group: 16800, version: 1}, {Group: "apiextensions.k8s.io", Version: "v1"}:              {group: 16700, version: 15}, {Group: "apiextensions.k8s.io", Version: "v1beta1"}:         {group: 16700, version: 9}, {Group: "admissionregistration.k8s.io", Version: "v1"}:      {group: 16700, version: 15}, {Group: "admissionregistration.k8s.io", Version: "v1beta1"}: {group: 16700, version: 12}, {Group: "scheduling.k8s.io", Version: "v1"}:                 {group: 16600, version: 15}, {Group: "scheduling.k8s.io", Version: "v1beta1"}:            {group: 16600, version: 12}, {Group: "scheduling.k8s.io", Version: "v1alpha1"}:           {group: 16600, version: 9}, {Group: "coordination.k8s.io", Version: "v1"}:               {group: 16500, version: 15}, {Group: "coordination.k8s.io", Version: "v1beta1"}:          {group: 16500, version: 9}, {Group: "auditregistration.k8s.io", Version: "v1alpha1"}:    {group: 16400, version: 1}, {Group: "node.k8s.io", Version: "v1alpha1"}:                 {group: 16300, version: 1}, {Group: "node.k8s.io", Version: "v1beta1"}:                  {group: 16300, version: 9}, {Group: "discovery.k8s.io", Version: "v1beta1"}:             {group: 16200, version: 12}, {Group: "discovery.k8s.io", Version: "v1alpha1"}:            {group: 16200, version: 9}, </del> <ins> {Group: "apps", Version: "v1"}:                               {group: 17800, version: 15}, {Group: "events.k8s.io", Version: "v1beta1"}:                 {group: 17750, version: 5}, {Group: "authentication.k8s.io", Version: "v1"}:              {group: 17700, version: 15}, {Group: "authentication.k8s.io", Version: "v1beta1"}:         {group: 17700, version: 9}, {Group: "authorization.k8s.io", Version: "v1"}:               {group: 17600, version: 15}, {Group: "authorization.k8s.io", Version: "v1beta1"}:          {group: 17600, version: 9}, {Group: "autoscaling", Version: "v1"}:                        {group: 17500, version: 15}, {Group: "autoscaling", Version: "v2beta1"}:                   {group: 17500, version: 9}, {Group: "autoscaling", Version: "v2beta2"}:                   {group: 17500, version: 1}, {Group: "batch", Version: "v1"}:                              {group: 17400, version: 15}, {Group: "batch", Version: "v1beta1"}:                         {group: 17400, version: 9}, {Group: "batch", Version: "v2alpha1"}:                        {group: 17400, version: 9}, {Group: "certificates.k8s.io", Version: "v1beta1"}:           {group: 17300, version: 9}, {Group: "networking.k8s.io", Version: "v1"}:                  {group: 17200, version: 15}, {Group: "networking.k8s.io", Version: "v1beta1"}:             {group: 17200, version: 9}, {Group: "policy", Version: "v1beta1"}:                        {group: 17100, version: 9}, {Group: "rbac.authorization.k8s.io", Version: "v1"}:          {group: 17000, version: 15}, {Group: "rbac.authorization.k8s.io", Version: "v1beta1"}:     {group: 17000, version: 12}, {Group: "rbac.authorization.k8s.io", Version: "v1alpha1"}:    {group: 17000, version: 9}, {Group: "settings.k8s.io", Version: "v1alpha1"}:              {group: 16900, version: 9}, {Group: "storage.k8s.io", Version: "v1"}:                     {group: 16800, version: 15}, {Group: "storage.k8s.io", Version: "v1beta1"}:                {group: 16800, version: 9}, {Group: "storage.k8s.io", Version: "v1alpha1"}:               {group: 16800, version: 1}, {Group: "apiextensions.k8s.io", Version: "v1"}:               {group: 16700, version: 15}, {Group: "apiextensions.k8s.io", Version: "v1beta1"}:          {group: 16700, version: 9}, {Group: "admissionregistration.k8s.io", Version: "v1"}:       {group: 16700, version: 15}, {Group: "admissionregistration.k8s.io", Version: "v1beta1"}:  {group: 16700, version: 12}, {Group: "scheduling.k8s.io", Version: "v1"}:                  {group: 16600, version: 15}, {Group: "scheduling.k8s.io", Version: "v1beta1"}:             {group: 16600, version: 12}, {Group: "scheduling.k8s.io", Version: "v1alpha1"}:            {group: 16600, version: 9}, {Group: "coordination.k8s.io", Version: "v1"}:                {group: 16500, version: 15}, {Group: "coordination.k8s.io", Version: "v1beta1"}:           {group: 16500, version: 9}, {Group: "auditregistration.k8s.io", Version: "v1alpha1"}:     {group: 16400, version: 1}, {Group: "node.k8s.io", Version: "v1alpha1"}:                  {group: 16300, version: 1}, {Group: "node.k8s.io", Version: "v1beta1"}:                   {group: 16300, version: 9}, {Group: "discovery.k8s.io", Version: "v1beta1"}:              {group: 16200, version: 12}, {Group: "discovery.k8s.io", Version: "v1alpha1"}:             {group: 16200, version: 9}, {Group: "flowcontrol.apiserver.k8s.io", Version: "v1alpha1"}: {group: 16100, version: 9}, </ins> // Append a new group to the end of the list if unsure. // You can use min(existing group)-100 as the initial value for a group. // Version can be set to 9 (to have space around) for a new group.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am testing API Priority and Fairness. I launched it like this: After the logging said the cluster is running and told me how to set KUBECONFIG, I did so in another shell and went looking for FlowSchema objects. But kubectl erroneously told me there is no such thing. However, looking in the kube-apiserver log shows flowschemas and prioritylevelconfigurations being read and written; for example: Also, I am able to read them with against the insecure port. Interestingly, the kube-apiserver log also includes this complaint: I will attach the kube-apiserver log. What you expected to happen: I expected to list them. How to reproduce it (as minimally and precisely as possible): I think the report above is pretty minimal. Anything else we need to know?: Environment: Kubernetes version (use ): this is the branch, git commit 916edd922e528f2f27542d3ca5967ca11fe153f8Cloud provider or hardware configuration: bare metal, intel, 16 "CPUs", 64 GB of memory OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux mjs-dev-1 4.15.0-74-generic -Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: hack/local-up- Network plugin and version (if this is a network-related bug): Others: Docker version 18.09.7 go version go1.13.4 linux/amd64
/sig api-machinery
/cc /cc /cc /cc
/assign
the "" API group is not included in the list of known groups to register at startup: That means resources inside that group will not appear in discovery, and clients like kubectl will not know how to find them.
You can see all resources visible via discovery by running
just as said, for now we have to manually create an apiservice into the cluster to get kubectl work.

i dont think we're adding this to the list until the beta stage, can we make an exception for the case?
It is fine to add alpha things to the list I linked so they get created automatically if needed. They only get registered if the relevant runtime-config is enabled.
Interrupts control plane upgrade in lazy scenario where it's assumed dns name isn't changed properly to active node or or single node control plane
oh cool actually it succeeds on the second main node but the kube apiserver pod doesnt start
the hacky way being to adjust that api creation based on this?
And I removed the flags too
Wait what is this? This was to my api manifest during kubeadm upgrade?


Question: } httpClient := &http.Client{} parsedNodeIP := net.ParseIP(nodeIP) <ins> protocol := utilipt.ProtocolIpv4 if parsedNodeIP != nil && parsedNodeIP.To4() == nil { glog.V(0).Infof("IPv6 node IP (%s), assume IPv6 operation", nodeIP) protocol = utilipt.ProtocolIpv6 } </ins> klet := &Kubelet{ hostname:                       hostname,

/kind bug What happened: Kubelet uses "iptables" instead of "ip6tables" in an ipv6-only cluster. This causes failed traffic for services (and probably a lot of other problems). For instance the chain is created in but is missing in ; When a is assigned the fails; What you expected to happen: All components should use in a ipv6-only cluster. should work in an ipv6-only cluster. How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Check iptables Optional; try a service type: LoadBalancer Anything else we need to know?: is hard-coded in finc in "" Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kvm VMs on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
/sig network
/area ipv6
Thanks for reporting this problem. I'll take a look.
is a link to the code with the hard-coded ipv4 protocol.
Thanks, I will send a pr to fix this issue.
/assign
I see this issue too. Also there is some related discussion with issue .
I just pulled the latest from master, and gave a try with IPv6 cluster, but I still see the error, when using kube-proxy + IPVS: E0829 16:51:56. 1 ] Failed to execute iptables-restore: exit status 2 (ip6tables-restore v1.4.21: The protocol family of set KUBE-LOOP-BACK is IPv4, which is not applicable. Were you able to try this with kube-proxy+IPVS?
Sorry, I didn't try ipv6 cluster with ipvs proxier. Those two issue are related, but not exactly the same issue. Seems we already have conclusion in , nobody working on that?


Question: // NamespaceSelector decides whether to run the webhook on an object based // on whether the namespace for that object matches the selector. If the // object itself is a namespace, the matching is performed on <del> // object.metadata.labels. If the object is other cluster scoped resource, // it is not subjected to the webhook. </del> <ins> // object.metadata.labels. If the object is another cluster scoped resource, // it never skips the webhook. </ins> // // For example, to run the webhook on any objects whose namespace is not // associated with "runlevel" of "0" or "1";  you will set the selector as

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: http.StatusSwitchingProtocols, ), ).Log() <ins> // monitor http requests var serverType string if s.auth == nil { serverType = "readonly" } else { serverType = "readwrite" } method, path, host := req.Method, trimURLPath(req.URL.Path), req.URL.Host longRunning := strconv.FormatBool(isLongRunningRequest(path)) servermetrics.HTTPRequests.WithLabelValues(method, path, host, serverType, longRunning).Inc() servermetrics.HTTPInflightRequests.WithLabelValues(method, path, host, serverType, longRunning).Inc() defer servermetrics.HTTPInflightRequests.WithLabelValues(method, path, host, serverType, longRunning).Dec() startTime := time.Now() defer servermetrics.HTTPRequestsDuration.WithLabelValues(method, path, host, serverType, longRunning).Observe(servermetrics.SinceInSeconds(startTime)) </ins> s.restfulCont.ServeHTTP(w, req) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: Attach          bool `json:"attach"` SELinuxRelabel  bool `json:"selinuxRelabel"` SupportsMetrics bool `json:"supportsMetrics"` <ins> FSGroup         bool `json:"fsGroup"` </ins> } func defaultCapabilities() *DriverCapabilities {

What happened: For some filesystems recursively applying an SELinux label through chcon or the GID permission is expensive (30 mins in some cases). There is better strategies for the recursive change and the call should be available through FLEX API. What you expected to happen: FLEX has a shell script call to label or set GID for a volume. Notes-- The fix should be backwards compatible.. If the shell call to perform the permission change is absent, then we fall back on the original method.


Question: CPUDetails: CPUDetails, }, nil } <ins> // getUniqueCoreID computes coreId as the lowest cpuID // for a given Threads []int slice. This will assure that coreID's are // platform unique (opposite to what cAdvisor reports - socket unique) func getUniqueCoreID(threads []int) (coreID int, err error) { err = nil if len(threads) == 0 { return 0, fmt.Errorf("no cpus provided") } if len(threads) != cpuset.NewCPUSet(threads...).Size() { return 0, fmt.Errorf("cpus provided are not unique") } tmpThreads := make([]int, len(threads)) copy(tmpThreads, threads) sort.Ints(tmpThreads) return tmpThreads[0], err } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CPU Manager uses topology from cAdvisor which uses which is different from in terms of coreID's. assigns coreID's in unique manner (platform-wise) whereas has coreID's from CPU registers which are unique only on socket level. This causes some unexpected cpu assignment on platforms with more than one socket. All tests written all "false positive" Example output: From What you expected to happen: Fix either by changing the way CPU manger sees topology (use ) or redesign CPU Manager to handle coreID's as socket only unique How to reproduce it (as minimally and precisely as possible): kubelet on dual socket node G pod that requests 2 cores pod gets 2 cores on different sockets
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: "type": "object" }, "io.k8s.api.flowcontrol.v1beta2.ResourcePolicyRule": { <del> "description": "ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) least one member of namespaces matches the request.", </del> <ins> "description": "ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) either (d1) the request does not specify a namespace (i.e., `Namespace==""`) and clusterScope is true or (d2) the request specifies a namespace and least one member of namespaces matches the request's namespace.", </ins> "properties": { "apiGroups": { "description": "`apiGroups` is a list of matching API groups and may not be empty. "*" matches all API groups and, if present, must be the only entry. Required.",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I looked at the comment on ResourcePolicyRule to understand exactly what its ClusterScope field means, and was dismayed to find an outdated precise description that does not mention that field. I expected that comment to be up to date. Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc /cc /cc /cc
/triage accepted


Question: "-A", string(kubeServicesChain), "-m", "comment", "--comment", fmt.Sprintf(`"%s loadbalancer IP"`, svcNameString), "-m", string(svcInfo.protocol), "-p", string(svcInfo.protocol), <del> "-d", fmt.Sprintf("%s/32", ingress.IP), </del> <ins> "-d", utilproxy.ToCIDR(net.ParseIP(ingress.IP)), </ins> "--dport", fmt.Sprintf("%d", svcInfo.port), )

We are working migrating ipv4 to ipv6, we should support ipv6 for IPVS. We should do the following things: remove ipv4 constraints of Node IPs() remove ipv4 in pkg/util/ipvs() use ipv6 conntrack tools use ipv6 iptables tools <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig network /assign /cc
/area ipv6
FYI: I've just posted a PR (PR ) to fix the way that kube-proxy was calling conntrack without the required "-f ipv6" CLI flag whenever conntrack was being executed for a service that's using IPv6 addresses. There were a couple of lines in that needed to be changed. I don't think I can add UT coverage for those lines until more support for IPv6 addresses is to Cc
ACK! I will take care of , please feel free to ping me when you want my hand :)
Sounds good, is all yours. :) Likewise, let me know if I can help.
Sure, mate.


Question: "fmt" "math" "math/rand" <ins> "net" "net/http" </ins> "os" "strconv" "sync"

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: func validateInitContainers(containers, otherContainers []core.Container, deviceVolumes map[string]core.VolumeSource, fldPath *field.Path) field.ErrorList { var allErrs field.ErrorList if len(containers) > 0 { <del> allErrs = append(allErrs, validateContainers(containers, deviceVolumes, fldPath)...) </del> <ins> allErrs = append(allErrs, validateContainers(containers, true, deviceVolumes, fldPath)...) </ins> } allNames := sets.String{}

I found that kubelet will not allocate hostports specified from initContainers. And PodFitsHostPorts in kube-scheduler does not take initContainer into consideration. Also, kube-apiserver think such pod is illegal, but I think it OK. We should make the behavior of spec.HostPorts in initContainer clear
/sig node
Exactly. I think this makes sense. Since initContainers are expected to be completed, we should not expose extra port mappings during creating pod sandbox, which may bring in security risks.
Sounds good to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // Ensure devices that should be detached/unmounted are detached/unmounted. for _, attachedVolume := range rc.actualStateOfWorld.GetUnmountedVolumes() { <del> if !rc.desiredStateOfWorld.VolumeExists(attachedVolume.VolumeName) { </del> <ins> // Check IsOperationPending to avoid marking a volume as detached if it's in the process of mounting. if !rc.desiredStateOfWorld.VolumeExists(attachedVolume.VolumeName) && !rc.operationExecutor.IsOperationPending(attachedVolume.VolumeName, nestedpendingoperations.EmptyUniquePodName) { </ins> if attachedVolume.GloballyMounted { // Volume is globally mounted to device, unmount it glog.V(12).Infof("Attempting to start UnmountDevice for volume %q (spec.Name: %q)",

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.