Question: // HasSupport checks if the given gvk supports the query param configured on v func (v *QueryParamVerifier) HasSupport(gvk schema.GroupVersionKind) error { <ins> if (gvk == schema.GroupVersionKind{Version: "v1", Kind: "List"}) { return NewParamUnsupportedError(gvk, v.queryParam) } </ins> oapi, err := v.openAPIGetter.OpenAPISchema() if err != nil { return fmt.Errorf("failed to download openapi: %v", err)

kubectl apply performs validation before the apply. Non-CRD resources (e.g. pods / deployments) don't need CRD schema for validation, but if we use resource, kubectl calls . In my Kubernetes cluster, administrator doesn't allow users to list CRDs, so user cannot apply non-CRD resources (e.g. pods / deployments) with resource. doesn't happen because we're going to apply two Deployments. should not happen because we're going to apply two Deployments. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted /assign
/priority backlog


Question: r.lastErrorTimeLock.Lock() defer r.lastErrorTimeLock.Unlock() d := time.Since(r.lastErrorTime) <del> if d < r.minPeriod { </del> <ins> if d < r.minPeriod && d >= 0 { // If the time moves backwards for any reason, do nothing // TODO: remove check "d >= 0" after go 1.8 is no longer supported </ins> time.Sleep(r.minPeriod - d) } r.lastErrorTime = time.Now()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: in func OnError() of ErrorHandlers in apimachinery it is expected to go on sleep for some time if not meet the minPeriod. However, if time happens hops to the past in the process. would be a negative number, thus, it may sleep a lot longer than we expected. What you expected to happen: The period for sleep should be reseted if time hops
/sig api-machinery
adding a check on whether d is < 0 should fix it, no? if d < 0, simply don't sleep.
ooh, we have time travelling kubes now :)
so if d < 0, it's better to sleep r.minPeriod.
i think Go 1.9 should fix it, as reported and verified in the other issue


Question: load( "@io_bazel_rules_go//go:def.bzl", "go_library", <ins> "go_test", </ins> ) go_library(

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: load( "@io_bazel_rules_go//go:def.bzl", "go_library", <ins> "go_test", </ins> ) go_library(

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: load( "@io_bazel_rules_go//go:def.bzl", "go_library", <ins> "go_test", </ins> ) go_library(

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Shows that prometheus.MustRegister will panic on things like dashes in the metric name. There are currently 138 calls to prometheus.MustRegister in the Kubernetes code base almost exclusively from controllers. We should audit the code to ensure that panic is the desirable response to a bad metric name for these calls vs. something like prometheus.Register which will instead return an error. What you expected to happen: See above How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: No Environment: All Kubernetes version (use ): 1.9 Cloud provider or hardware configuration**: All OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I did an analysis on this issue, and this is what I found: of the calls use a static metric name (hardcoded at the method invocation via a string and/or constant); is only a single case where the metric name is constructed dynamically (from a parameter), at the file (and happens to be where panicked); My proposal is: a. For static metric names (1), leave them as they are; b. For dynamic metric names (2), where the caller might not know that a parameter will be used to construct a metric name: Update the file to use instead of , and return the error if the metric cannot be registered (already submitted PR ); A quick search from the method invocation (from the above file) results in 17 invocations, mostly controllers, but although returns an error, no one of those 17 results is catching the error. So, we need to update all invocations to catch the error (I plan to submit new PRs to address this).

/sig instrumentation
We can trace all the task for update metrics.RegisterMetricAndTrackRateLimiterUsage invocations, and I am very happy to help fix it. [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [ ] [ ] [x] [x] [x]
Please go ahead with the ones not yet updated if you want.
OK, thanks!
now, we catch the error and when there return an error, we return in the function . But in the test file we use function for many times and Then there will be a lot of error in the test file, such as: then the test will always go fail. I think we should distinguish the error and when the error is Duplicate registration we dhould ignore it and return the normal controller.
Can you please provide an example where this is happening? I ran the tests on several controllers and haven't found that error. Ignoring this error doesn't seem a good idea IMO, if there're some tests that fail, we should investigate why are failing and fix them appropriately.
Sorry, I thought you were talking about the unit tests, didn't see the integration tests problem. I've submitted to fix this and ran the integration tests to check that creating multiple controllers don't fail anymore.
actually in some cases the unit tests is also failed such as endpoint controllerreplicaset controller. because different set up before test. So we should take a action to resolve it.


Question: load( "@io_bazel_rules_go//go:def.bzl", "go_library", <ins> "go_test", </ins> ) go_library(

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to use  in a unit test, however I get a nil pointer panic when is invoked. What you expected to happen: would return a that doesn't panic on any method call. Ideally, it would allow me to interact with the client in someway for testing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: client-go version: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
I believe this also impacts returning logs via a reactor as does not implement
Seems to be a problem with . Since we're creating the request with all empty fields. Not sure what the best way it to resolve this. The best thing I came up with is to use to return a request with everything set up. But I'm a noob here so that's just a shot in the dark :man_shrugging:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The request returned has a nil which is used in the
I have filed a PR trying to fix this issue.


Question: "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apiserver/pkg/registry/generic" genericregistry "k8s.io/apiserver/pkg/registry/generic/registry" "k8s.io/apiserver/pkg/registry/rest"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The api bind a pod to a node. But the pod is specified by name (not uid), which is not a unique identifier for pod. So the caller of can't make sure the pod is exactly what he want. api can bind a pod with specified uid to a node. If the uid not match, return an error a pending pod by statefulset named busybox, make sure it's pending, the scheduler will not bind it(use non-existed node selector or other method). So we will get a pending pod busybox-0. following code the in the code, delete the pod busybox-0 manually during the sleep or other method. The statefulset busybox will create another pod busybox-0 with different uid , the api in the code above will bind an unexpected pod to node The logic in scheduler is the same. If the pod is recreate with the same name and different resource require or selector between scheduler's and , scheduler will do a wrong bind for the pod. I think the root cause is the api does not check the uid. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-20T09:01:41Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-aliyunedge.1", GitCommit:"", GitTreeState:"", BuildDate:"2021-03-08T09:56:01Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig scheduling for triage (the binding feature of (pod to node) is for scheduling).
For the sake of precedence, are there other APIs that are based on object UID?
uids are used as a precondition is used in many places. if the server started checking that a uid on the binding object (if specified) matched the uid of the pod being modified, that seems sufficient
is there any way for the caller of Bind to confirm that he is binding the right pod? or not uid, use resource version?
that would require: the server to check that bind UID (if specified) matches the pod UID (passing the binding uid from BindingREST#Create → assignPod → setPodHostAndAnnotations and verifying the uid match in setPodHostAndAnnotations) the binding API caller to set in the posted binding object (as illustrated in the description of this issue)
actually, kube-scheduler(the most common binding API caller) has set the in the posted binding object. but the server doesn't check the uid. in my program, i do those to work around(not elegant, but it works...): call the binding API with an uid in annotation(i found that BindingREST#Create → assignPod → setPodHostAndAnnotations will carry annotation). check the matchs , if not match, delete the pod and wait for an new pod ... from your point of view, is it necessary to ensure the pod is the caller want when someone call bind API? what's your advice for this issue
given the scheduler made decisions based on the current shape of the pod spec, I think it makes sense for it to work to make sure it is binding that instance of the pod I would advocate for supporting uid as a precondition in the binding API and starting to use it from the scheduler
i agree with you
/remove-sig api-machinery
I also agree with this and this cannot achieve only on sig/scheduling. Sorry, if it is not suitable, but I'd like to add sig/api-machinery label again.

/sig api-machinery
Either scheduling or node sigs own the pod binding API. Either of those would be a good place to coordinate adding a fix to honor uid as a precondition API machinery owns the generic API plumbing and API server behavior, not implementations of most specific APIs like the pod or pod binding APIs /remove-sig api-machinery
Given you already exercised Jodan's suggestion locally, would you mind raising an offical PR?
how about this PR


Question: DialOptions:          dialOptions, Endpoints:            c.ServerList, TLS:                  tlsConfig, <ins> Logger:               etcd3ClientLogger, </ins> } return clientv3.New(cfg)

The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact.

What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).


Question: <ins> apiVersion: v1 kind: ReplicationController metadata: name: test-rc labels: name: test-rc spec: replicas: 1 template: metadata: labels: name: test-rc spec: containers: - name: test-rc image: nginx ports: - containerPort: 80 </ins>

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: if !reflect.DeepEqual(stateData, item.wantStateAfterPreFilter) { t.Errorf("state got after prefilter does not match: %v, want: %v", stateData, item.wantStateAfterPreFilter) } <del> t.Logf("call Filter and check status") </del> <ins> t.Logf("Verify: call Filter and check status") </ins> gotStatus := p.Filter(ctx, state, item.pod, nodeInfo) if !reflect.DeepEqual(gotStatus, item.wantFilterStatus) { t.Errorf("filter status does not match: %v, want: %v", gotStatus, item.wantFilterStatus)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: The unit test Which test(s) are flaking: Testgrid link: Reason for failure: In volume scheduling, we maintain internal caches that watch for PV/PVC events. However, we don't have a method to wait for them to be synced in UT. When the cache in the informer factory is synced but the execution of registered handlers may not finish. <!-- In production, this is not a problem. Even if the scheduler marks the pod uncacheable due to the stale cache in PVAssumeCache/PVCAssumeCache, the pod will be moved to the active queue later as the scheduler . related: --Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig scheduling /sig storage
Even if I check the PV/PVC assume cache and wait for PV/PVC objects to be into the cache, the PV/PVC objects may not be . It seems that the shared informer with the fake client has a bug that registered event handlers may not be called (or handlers are not registered successfully).
The share informer with the fake client may miss events of objects which are created before caches are synced. it's better to create testing objects before starting the informers
Although the event handlers may not finish when the caches were synced in theory, however, it seems it's not necessary to wait for PV/PVC assume cache to be synced practically. I've run a thousand times (parallel: 10) with locally and no failures.


Question: ) // SystemPriorityClasses defines special priority classes which are used by system critical pods that should not be preempted by workload pods. <ins> // NOTE: In order to avoid conflict of names with user-defined priority classes, all the names must // start with scheduling.SystemPriorityClassPrefix which is by default "system-". </ins> var SystemPriorityClasses = map[string]int32{ "system-cluster-critical": SystemCriticalPriority, "system-node-critical":    SystemCriticalPriority + 1000,

In a recent discussion on , suggested to keep priority class names that start with "system-" for system use only and not allow user defined priority classes to start with this prefix. Given that we have reserved priority values 1 Billion for system use only, it makes sense to reserve a prefix as well to ensure that predefined priority classes won't conflict with user defined ones in the future. Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig scheduling /assign ref


Question: import ( "fmt" <ins> "reflect" </ins> "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api"

We have seen that on VirtualBox in particular kubelet picks first interfaces, which is usually NATed and is of no use. In particular the issue comes up with service IP, which gets mapped to endpoint on the NATed interface.
If this is still a problem, can you explain more - I don't quite understand - and suggest heuristics for better solutions?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "net" "net/http" "regexp" <ins> "sort" </ins> "strings" "k8s.io/api/core/v1"

/kind bug /sig network /assign What happened: For events being generated within the GCE cloudprovider, slices of firewall parameters are not being sorted before converted into a string. Due to random ordering, this may result in the same line being repeated.


Question: } return nil } <ins> func isNamespaced(gvr schema.GroupVersionResource, discoveryClient discovery.DiscoveryInterface) (bool, error) { if gvr.Resource == "*" { return true, nil } apiResourceList, err := discoveryClient.ServerResourcesForGroupVersion(schema.GroupVersion{ Group: gvr.Group, Version: gvr.Version, }.String()) if err != nil { return true, err } for _, resource := range apiResourceList.APIResources { if resource.Name == gvr.Resource { return resource.Namespaced, nil } } return false, fmt.Errorf("the server doesn't have a resource type '%s' in group '%s'", gvr.Resource, gvr.Group) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: recorder), NewRealStatefulSetStatusUpdater(kubeClient, setInformer.Lister()), history.NewHistory(kubeClient, revInformer.Lister()), <ins> recorder, </ins> ), pvcListerSynced: pvcInformer.Informer().HasSynced, queue:           workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "statefulset"),

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: package pidlimit import ( <ins> "fmt" </ins> "io/ioutil" "strconv" <ins> "strings" </ins> "syscall" "time" <del> "k8s.io/apimachinery/pkg/apis/meta/v1" </del> <ins> v1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> statsapi "k8s.io/kubelet/pkg/apis/stats/v1alpha1" )

Condition PIDPressure is not set to true when the system thread number exceeds even it's reaching the limit. Condition PIDPressure is set to true correctly See cause below The bug is due to , which truncates any process number greater than because of type conversion. See kernel code . Reading it from should be working. v1.23.1 <detailsNA </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-longterm


Question: } pl, ok := obj.(*example.PodList) if !ok { <del> t.Fatalf("Expected PodList but got %t", pl) </del> <ins> t.Fatalf("Expected PodList but got %v", pl) </ins> } if len(pl.Items) != 3 { t.Errorf("Expected PodList of length 3 but got %d", len(pl.Items))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Unit tests are failing with the latest golang(1.12) What you expected to happen: Tests should pass without any fauilures. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery


Question: {% endif %} {% endif %} - logrotate <del> {% if grains['cloud'] is defined and grains['cloud'] == 'azure' %} - openvpn-client {% else %} - sdn {% endif %} </del> - monit 'roles:kubernetes-master':

On latest pull, the Vagrant kube- scripts completes successfully, but the Salt setup in the minion keeps going forever. End result is a cluster that is not useable. Looking at '/var/log/salt/minion' on the minion box, see lots of entries like: 2015-03-04 20:09:11,952 [ ][INFO ] Starting a new job with PID 2015-03-04 20:09:11,954 [ ][INFO ] Returning information for job: 2015-03-04 20:09:12,027 [ ][DEBUG ] Decrypting the current master AES key 2015-03-04 20:09:12,028 [ ][DEBUG ] Loaded minion key: 2015-03-04 20:09:12,217 [ ][INFO ] User vagrant Executing command with jid 2015-03-04 20:09:12,217 [ ][DEBUG ] Command details {'tgt_type': 'glob', 'jid': '', 'tgt': '*', 'ret': '', 'user': 'vagrant', 'arg': [], 'fun': ''} No other help looking at the master's salt log. This is Vagrant 1.7.2, with VirtualBox 4.3.20 on Mac OSX (Yosemite).
Is this repeatable for you if kube-down and kube-up?
Yes, this is reproducible on my laptop. But, I was able to get it up by doing a kube-push after the failed kube-up. , Eric Tune wrote:
Self assigning and will try to reproduce. Sent from my iPhone
I was able to reproduce. It appears that the Kubelet fails to start properly on first start:
The push basically caused the Kubelet to restart, and then it does work. Without the push, systemd would have eventually restarted the Kubelet. We need more investigation as to what was causing the Kubelet to fail on initial startup.


Question: "finalizer.go", "io_util.go", "metrics.go", <ins> "nested_volumes.go", </ins> "resize_util.go", "util.go", ] + select({

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: fi else case "${KUBERNETES_PROVIDER}" in <del> gce|gke) </del> <ins> gce|gke|kubemark) </ins> files="${files} ${gce_logfiles}" ;; aws)

It seems that we stopped uploading logs to GCS around Friday/Saturday. As examples, this is the first kubemark-scale run that doesn't have logs: But we also don't have any logs e.g. for kubernetes-gce-e2e suite (which seems to be our main suite), e.g.: This seems to be the last run, for which we have logs: which suggests that it was broken around 2016-10-08 01:00:00
P0 - because it makes any debugging roughly impossible.
is working on this
It seems that after fixes from yesterday, we are still lacking logs from all kubemark runs.
Lack of kubemark logs is significantly slowing down any scalability-related work, which is our P0 goal for this quarter, so please prioritize it. - FYI ^^
As an example: [is a run from an hour ago] and it contains only file.
I'm going to migrate this job over to where our code (rather than a jenkins plugin) captures and uploads logs, which we want to do anyway and will hopefully resolve this. Odd... if we look at job/kubernetes-e2e-gce/configure we will a bunch of conditional build steps in the post-build actions configuration, which matches what is specified in yaml: We also have this specified in kubemark yaml: However I don't see where this is applied in the actual configuration: /job/kubernetes-kubemark-gce-scale/configure
I suspect that the job updater ran at some point this weekend while the conditional build step plugin wasn't working, so it wasn't able to add that step. I'm not sure why it hasn't rectified it since.
That seems to be the problem. I've cleared the job cache and done a full rebuild, so gce-scale and anything else that was updated why the postbuild plugin was missing should be fixed now.
Hmm - it doesn't seem to be fully fixed. Basically, we used to have logs from kubemark master machine, e.g.: (there is this kubemark-500-kubemark-master/ directory) However, even now (this is the last run), we no longer have those logs: So something is still broken...
Any thoughts about it? The logs are still missing, e.g.:
All the files in _artifacts are being properly uploaded.

Something else is failing to copy the logs onto the node. In In , it seems to be copying more: Did something else change in kubemark between these time periods? Is scp from the master broken now? This is a separate issue.
I'm not aware of any other changes. - we should try to debug it tomorrow.
We did touch recently in . Might be related.
Yeah, I was just trying to work out if it was. I tried to keep the fake path alive as best I could.
- if you could figure out if it was because of your PR (timing matches) that would be great...
I confirm that this is fixed now. - thanks a lot for fixing this!


Question: } func handleAttachPod(f cmdutil.Factory, podClient corev1client.PodsGetter, ns, name string, opts *attach.AttachOptions) error { <del> pod, err := waitForPod(podClient, ns, name, podRunningAndReady) </del> <ins> pod, err := waitForPod(podClient, ns, name, opts.GetPodTimeout, podRunningAndReady) </ins> if err != nil && err != ErrPodCompleted { return err }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: "involvedObject.namespace": f.Namespace.Name, "reason":                   "Failed", }.AsSelector().String() <del> err = framework.WaitTimeoutForPodEvent(f.ClientSet, pod.Name, f.Namespace.Name, selector, errorMsg, framework.PodEventTimeout) </del> <ins> err = framework.WaitTimeoutForPodEvent(f.ClientSet, pod.Name, f.Namespace.Name, selector, errorMsg, framework.PodStartTimeout) </ins> Expect(err).NotTo(HaveOccurred(), "while waiting for failed event to occur") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: From , it took around a minute for the CSI plugin to register. It would be good to understand why it took so long and if it's normal/expected. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
is this a blocker for 1.13? do we have a baseline of how long its expected to take for the registration?
I took a look at this. Looks like the external sidecar causes the delay (see ): So, failed to connect to CSI driver, and took a minute to fail. Here are the logs from the host path CSI driver side (see ): So it looks like the tries to open a connection a split second before the CSI hostpath driver starts listening: And the takes a minute to fail/recover from that. Based on this, the issue does not seem to be due to any code in . And is not a blocker for 1.13. I will open an issue on the driver-registrar to track if this delay can be reduced.
Opened Closing this.
Good investigation! Then, should we add some codes to check if driver-registrar complete driver registration with more than 1 minute timeout? IIUC, tests may flake by trying to attach volume before driver become ready in any tests, unless we don't add this kind of logic. To achieve it, I think that we would be able to check string from driver-registrar pod by the same way to , at the end of each driver's .
Can we try the grpc options used in other spots? Looks like the current options for the connect are here: cadvisor for example has and
will do!


Question: registry                = initReg() dockerLibraryRegistry   = registry.DockerLibraryRegistry e2eRegistry             = registry.E2eRegistry <ins> e2eGcRegistry           = "gcr.io/kubernetes-e2e-test-images" </ins> gcAuthenticatedRegistry = registry.GcAuthenticatedRegistry gcRegistry              = registry.GcRegistry googleContainerRegistry = registry.GoogleContainerRegistry

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: All sig-windows jobs (master): Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this PR merged: Testgrid link: Reason for failure: The mentioned PR removed hardcoded image names from tests, centralizing them into , but there were a few omissions: entry was written for. registry set for ` is because that registry contains the Windows images needed for all the Kubernetes E2E tests. Anything else we need to know: /sig testing /sig windows /area conformance /priority important-soon


Question: // if usage shows no change, just return since it has no impact on quota deltaUsage := evaluator.Usage(inputObject) if admission.Update == op { <del> prevItem, err := evaluator.Get(namespace, name) if err != nil { return nil, admission.NewForbidden(a, fmt.Errorf("Unable to get previous: %v", err)) </del> <ins> prevItem := a.GetOldObject() if prevItem == nil { return nil, admission.NewForbidden(a, fmt.Errorf("Unable to get previous usage since prior version of object was not found")) </ins> } prevUsage := evaluator.Usage(prevItem) deltaUsage = quota.Subtract(deltaUsage, prevUsage)

the ability to get the previous version of an object directly via admission control. As a result, admission code can now use this rather than fetching the object directly. Need to update:
/cc - this is a necessary follow-up, will get to it soon.


Question: - group: root - mode: 755 <ins> /var/lib/kubelet/pki: file.directory: - mode: 755 - makedirs: True </ins> # The default here is that this file is blank. If this is the case, the kubelet # won't be able to parse it as JSON and it will not be able to publish events # to the apiserver. You'll see a single error line in the kubelet start up file

https://k8s- Judging by the timestamp, it's this batch of merge caused the failure: NONE
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
looks like it's your pr causing the failure ^^^
cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
It's still failing, See build log: https://k8s-, kubelet log: error in kubelet log:


Question: command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <del> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } if mounter.withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", target) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) </del> <ins> return checkUmountError(target, command, output, err, mounter.withSafeNotMountedBehavior) </ins> } return nil }

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: execAffinityTestForLBService(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: provisioner: kubernetes.io/gce-pd parameters: type: pd-standard <ins> allowVolumeExpansion: true </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Enable it in the default storageclass. Why is this needed: Since resizing behavior has to be triggered by user action, I don't see a reason why we shouldn't allow it by default. /assign
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: echo "${KUBE_VERSION}" else echo "Using version at ${1}: ${KUBE_VERSION}" >&2 <del> tars_from_version curl --fail -o kubernetes-${KUBE_VERSION}.tar.gz "${KUBE_TAR_URL}" </del> <ins> if [[ ${KUBE_VERSION} =~ ${KUBE_VERSION_REGEX} ]]; then curl --fail -o kubernetes-${KUBE_VERSION}.tar.gz "https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes.tar.gz" elif [[ ${KUBE_VERSION} =~ ${KUBE_CI_VERSION_REGEX} ]]; then curl --fail -o kubernetes-${KUBE_VERSION}.tar.gz "https://storage.googleapis.com/kubernetes-release/ci/${KUBE_VERSION}/kubernetes.tar.gz" else echo "Version doesn't match regexp" >&2 exit 1 fi </ins> fi

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: optional bool readOnly = 3; } <del> // Binding ties one object to another. // For example, a pod is bound to a node by a scheduler. </del> <ins> // Binding ties one object to another; for example, a pod is bound to a node by a scheduler. // Deprecated in 1.7, please use the bindings subresource of pods instead. </ins> message Binding { // Standard object's metadata. // More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata

Forked from We need to update the scheduler (ours and mesos) to use pods/binding instead of /binding and then we can delete /binding.
cc
This is about the top-level resource, not the subresource.
This is where mesos is using binding as top level resource: We need to change that to Resource("pods").Subresource("bindings")
/cc
If we turn this off in 1.2, will that be a problem for you?
Did we announce we were going to remove this API? Don't we need to have a process for taking it away?
Yes it would be problematic unless our scheduler was updated to use whatever was replacing it. AFAICT even the core scheduler is still using /bindings: , Clayton Coleman wrote:
Kicking this can down the road.
We should mark this deprecated.
According to , kube-mesos refactor will use HTTPExtender, so it's a not big issue to kube-mesos. But for k8s, I think scheduler is using API now ().
/assign
/close I'm going to close this; if any comments, please let me know.
We need to actually mark the endpoint deprecated so we can remove it.
OK, got that; need to deprecate it for other user. I think k8s can handle it to use .
, can you help to remove ? it's not related with Mesos anymore. I'll create a PR to deprecate .
Reopening since that PR just a deprecation warning. We still need to remove the API.
wow, agree. , would you help to remove milestone? We will not remove API in 1.7 or 1.8; but I'll trace that.
Done
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
18 still have.
21 still have the resource. Is anyone already working on it or is there a reason why the binding resource still get served?

not entirely sure if i'm right, but from my current assumption it seems that there isn't an implementation for the for pods (xref: ) and we still use the generic resource for pods (). The resource is deprecated since 1.7 but i don't see any further progress :thinking: () dump of an :
what is the progress with this issue? Is it a backlog item that is tracked by sig scheduling?
I'm not aware of this issue. We will put it into our sig meeting to discuss.
we went through this in today's scheduling meeting. We're ok with shifting v1.Binding to pod's subresource. Before acting on it, there are several factors we need to consider: impact to external integrators - either 3rd-party scheduler or scheduler plugins (using extension point). Follow the conventional API deprecation/removal guide/process to ensure a smooth migration. overall volume of efforts. In addition to scheduling efforts, are there any API Server side efforts that need to be tweaked as well? we need to get +1 from
Is there a summary of why moving this API is necessary given that is has been around for quite sometime now? Is it related to RBAC?
/binding is a v1 API, and as such, deletion is not permitted. It can be marked deprecated, and users pointed to pods/binding by documentation as the preferred method, but not removed.
It's more to pay the technical debt. Given the action "binding" is performed against pods, composing a pod/binding instead of v1.Binding makes more sense. Thanks for clarification we cannot remove v1 APIs.
ok, thanks for the clarification, so this is not enabling a new use case, only a cleanup.
Right, and due to the stable API compatibility guarantees, I don't think it's a cleanup we can do. /close
Closing this issue. <detailsIn response to : so this is not enabling a new use case, only a cleanup. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry for the late comment on this but i was AFK for a longer time. I'm not sure if we missed something. Currently there is only the implementation.

via the the API endpoint exists but it refers to the only existing kind (could be checked via ): As there isn't a seperate implementation for IMO we should remove the deprecation warning as we don't have an alternative implementation for that in place. / WDYT?


Question: glog.Errorf("iscsi: last error occurred during iscsi init:n%v", lastErr) } <del> //Make sure we use a valid devicepath to find mpio device. devicePath = devicePaths[0] for _, path := range devicePaths { // There shouldnt be any empty device paths. However adding this check // for safer side to avoid the possibility of an empty entry. if path == "" { continue } // check if the dev is using mpio and if so mount it via the dm-XX device if mappedDevicePath := b.deviceUtil.FindMultipathDeviceForDevice(path); mappedDevicePath != "" { devicePath = mappedDevicePath break } </del> <ins> // Try to find a multipath device for the volume if 1 < len(bkpPortal) { // If the PV has 2 or more portals, wait up to 10 seconds for the multipath // device to appear devicePath = waitForMultiPathToExist(devicePaths, 10, b.deviceUtil) } else { // For PVs with 1 portal, just try one time to find the multipath device. This // avoids a long pause when the multipath device will never get created, and // matches legacy behavior. devicePath = waitForMultiPathToExist(devicePaths, 1, b.deviceUtil) } // When no multipath device is found, just use the first (and presumably only) device if devicePath == "" { devicePath = devicePaths[0] </ins> } glog.V(5).Infof("iscsi: AttachDisk devicePath: %s", devicePath)

/kind bug /sig storage What happened: When mounting iSCSI PV kubelet doesn't give multipathd enough time to create multipath device and fallbacks to mounting single SCSI device What you expected to happen: it uses multipath device to mount How to reproduce it (as minimally and precisely as possible): create ~5 pods on a single node, each with dynamically provisioned PV (we use Trident for that), observe how some mounts are /dev/sd and not /dev/mapper/ Environment: Kubelet 1.9.3 Baremetal + NetApp ONTAP ove iSCSI + Trident dynamic volume provisioner CoreOS (kernel 4.14.24, multipath-tools 0.6.4)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can reproduce this issue fairly often (although not reliably). The problem is in in the AttachDisk() function. The basic problem is that the method that finds the multipath device looks at each device path one time, and if none of them have an associated multipath device, it just falls back to formatting and mounting the first device path. There is no wait loop for the multipath device to appear. It would be easy to add a wait loop, but that would slow down the non-multipath use case. A possible optimization would be to only wait when 2 or more paths are known, but that still opens up the possibility of incorrect behavior when attaching while all but one paths are down, or when only one path is discovered (for whatever reason).
After thinking about this over lunch, I think a possible fix would be: add a 10 second wait loop for the multipath device if and only if the PV are 2 or more target portals specified. This puts control in the hands of the provisioner without modifying the API at all, and not breaking backwards compatibility. The one corner case I can see is what would happen the PV has 2 portals, but the system is only able to log in to one.

This would occur when multipath was correctly configured on the storage controller and in the provisioner, but one of the paths was down (for whatever reason) at attach time. I'm inclined to make the attachment FAIL if the multipath device can't be found within 10 seconds, which would be a change to current behavior, but this can easily be worked around by the cluster administrator setting up multipathd to create devices for single path disks. Provisioners would want to document the recommended way to do that (i.e. set find_multipaths = no, and configure the blacklist appropriately).
is vaguely described as "target portal ips for high availability", without describing what the "high availability" means. Should we assume that it means multipath? Then Kubernetes can always wait for multipath device to appear when more that one is specified. Would it break anything? Does anyone run multiple portals for the same volume without multipath? Is it something we should support? any opinion?
yes, the iscsi portals were made possible by for HA/mpio use case (see ). The lack of concrete description makes it less obvious though.
was it working perfectly with older kube versions and the issue started to pop up recently or from ? Why I am asking is, multipath support was in kube () and atleast I havent seen this issue reported till this report or very recently. Gluster team were also using multipathing feature for quite some time where we always got mpath device mounted reliably, but noticed this issue very recently or with latest OCP builds. iic, There are good number of changes happened in between especially on device discovery/attach ( ex: ) disk paths. I am not concretely saying its a regression or a behavioral change or side effect of some other patch, but trying to figure out if thats the case. any thoughts here ? my understanding was/is that with more target portal IPs means ( atleast) in ISCSI context, but we could have mention as well in the description. Suppose the backup portal IPs are filled by provisioner/admin but initiator havent enabled ( dm_mulitpath is not loaded or deamon service is not running) software in the kubelet/node, what will happen ? Is it going to be infinite wait for failure ?

my understanding is that, if multipath is enabled, even if it failed ( for some reason) for backup paths, a mpath device will be created for single path in the system. If thats the case above shouldnt be a concern and device mapper layer will take care of the path discovery later when path becomes stable/available. I dont think so.
we started with 1.9.x on kubelets, so can't tell much about mounting behaviour of previous versions. it is going to fail, but it is fine, nodes should be correctly configured. Lets say you specify fsType, but given FS module is not loaded by kernel, or now mkfs.<FSbinary is available to the kubelet, mount is going to fail and thats OK IMHO. Kubelet shouldn't try to fallback to anything less than was asked, mounting via single portal when multiple portals are specified is similar to trying "default FS" if required fsType can't be used. Yes, default configuration for multipathd is option (note, that default in Linux distributions might be different), which creates multipath device for every non-blacklisted block device.
I'm fairly confident that adding a wait loop for the multipath device in the case when there are 2 or more portals in the PV spec won't cause any regressions. The thing that gives me pause is that there are cases when there's a single portal in the PV spec, but the device still supports multipath and the existing sendtargets command is the mechanism for finding those other paths. I'm wondering if it's possible to detect that case (reliably) and use a wait loop for the multipath device in those cases too.
is the default on RHEL 7, see
Did you mean Multiple Connection Per Session (MC/S)? I don't think this is supported in open-iscsi. Open-iSCSI only supports MPIO (multiple portals)
I don't understand the move by RedHat to default to findmultipaths = no. It basically guarantees bad behavior in cloud-oriented use cases when there's a temporary path failure. My reading of the linked document is that the main motivation for changing the default is to avoid the need to setup a correct blacklist. It seems to me that correct behavior is more important that avoiding some work up front when configuring multipath. No, I'm referring to the .

If a storage backend has multiple target portals, but the dynamic provisioner only puts one portal in the PV spec, then as long as that one portal is accessible at attach time, open-iscsi will discover the other portals and log into them. The only weakness to this approach is what happens when that one path is inaccessible at attach time due to a network switch/network port/NIC failure, while the other ports are still available.
, AFAIK multipathd (binary) in RHEL 7 defaults to , but configuration utility they ship writes config with , that utility needs to be called explicitly , IMHO that should be direct scsi mount unconditionally, without multipath involved
The thing that gives me pause is that there are cases when there's a single portal in the PV spec I would be inclined to agree, except that someone the sendtargets logic to the iscsi code. This code exists for no reason other than to allow finding multiple path given a single path. In fact the existing NetApp dynamic provisioner (Trident) relies on that behavior currently to get multipath support (this is something I've fixed and will upstream shortly). If we decide that PVs with exactly one portal should not support multipath, then we should remove that sendtargets logic from the iscsi module in kubelet.


Question: RUN apt-get -q update &&  apt-get install -y dnsutils &&  apt-get clean <ins> ADD https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_linux_BASEARCH.tgz /coredns.tgz RUN tar -xzvf /coredns.tgz && rm -f /coredns.tgz </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should support configurable pod Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Since ` was used instead. Anything else we need to know: /sig testing /sig windows


Question: echo "COPY nsswitch.conf /etc/" >> "${docker_file_path}" fi <del> "${DOCKER[@]}" buildx build  </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" buildx build  </ins> --platform linux/"${arch}"  --load ${docker_build_opts:+"${docker_build_opts}"}  -q

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: The overall job Since when has it been failing: Sometime between Jan 29 07:00 PST (commit ) and 16:13 PST (commit ) There are some runs that could not be scheduled on a node between these two runs, so it's not clear exactly when it started failing Testgrid link: https://k8s- Reason for failure: And then it spits out all the possible Docker options and commands. Anything else we need to know: Feels like it may be related to ? -- That was merged Jan 28 though, and there were successful runs between that and this run Spyglass example for failure: Contrast with Spyglass example for last success: /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I confirm that I hit the same issue in a CI that builds kubernetes using
is removed in docker 20.10?
Looking into this /assign xref
xref /cc
Yeah docker buildx is not initialized within this job. Like mentioned, we probably broke other CI's too. I'm wondering if a init script would help us here, but I'm just trying to find out what the technical differences between the and are.
/assign


Question: # Set up the context directory for the kube-build image and build it. function kube::build::build_image() { mkdir -p "${LOCAL_OUTPUT_BUILD_CONTEXT}" <ins> # Make sure the context directory owned by the right user for syncing sources to container. chown -R ${USER_ID}.${GROUP_ID} "${LOCAL_OUTPUT_BUILD_CONTEXT}" </ins> cp /etc/localtime "${LOCAL_OUTPUT_BUILD_CONTEXT}/"

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2850+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-06T09:06:04Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2753+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-04T08:46:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} What happened: How to reproduce it (as minimally and precisely as possible): the owner of file is not right
Do you need to be building as root?
I get an annoying message when building as normal user: So I run as root. There will return a error if the outpt build directory is not owned by root when running as root. I think we could use to fix it.
I hope to get some of your suggestions
did you previously build as ? you might need to before a build as a normal user works again.


Question: repeated NetworkPolicyEgressRule egress = 3; // List of rule types that the NetworkPolicy relates to. <del> // Valid options are Ingress, Egress, or Ingress,Egress. </del> <ins> // Valid options are "Ingress", "Egress", or "Ingress,Egress". </ins> // If this field is not specified, it will default based on the existence of Ingress or Egress rules; // policies that contain an Egress section are assumed to affect Egress, and all policies // (whether or not they contain an Ingress section) are assumed to affect Ingress.

Is this a BUG REPORT or FEATURE REQUEST?: bug report /kind bug What happened: I was reading the documentation for as defined What you expected to happen: To have clear documentation How to reproduce it (as minimally and precisely as possible): Read the following, as seen in the 1.12 docs: Anything else we need to know?: Nope Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is it a case whereby only can be specified as , or only , or both (note the ) ? If that's the case, do you think the following would make it clearer?
It is confusing to have commas used for English grammar and array element delineation in the same sentence. What about simply this?
sounds like a great fix! Putting out a quick pr with this change now :)


Question: // can short circuit loop (break) if oldest requests are not timing out // as newer requests also will not have timed out <del> // now - requestWaitLimit = waitLimit waitLimit := now.Add(-qs.qCfg.RequestWaitLimit) </del> <ins> // now - requestWaitLimit = arrivalLimit arrivalLimit := now.Add(-qs.qCfg.RequestWaitLimit) </ins> reqs.Walk(func(req *request) bool { <del> if waitLimit.After(req.arrivalTime) { req.decision.Set(decisionReject) timeoutCount++ metrics.AddRequestsInQueues(req.ctx, qs.qCfg.Name, req.fsName, -1) req.NoteQueued(false) </del> <ins> if arrivalLimit.After(req.arrivalTime) { if req.decision.Set(decisionReject) && req.removeFromQueueLocked() != nil { timeoutCount++ req.NoteQueued(false) metrics.AddRequestsInQueues(req.ctx, qs.qCfg.Name, req.fsName, -1) } </ins> // we need to check if the next request has timed out. return true } // since reqs are sorted oldest -> newest, we are done here. return false }) // remove timed out requests from queue if timeoutCount > 0 { <del> // The number of requests we have timed out is timeoutCount, // so, let's dequeue the exact number of requests for this queue. for i := 0; i < timeoutCount; i++ { queue.requests.Dequeue() } // decrement the # of requestsEnqueued </del> qs.totRequestsWaiting -= timeoutCount qs.obsPair.RequestsWaiting.Add(float64(-timeoutCount)) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned .

Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to .

hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .


Question: func TestValidatePodSpec(t *testing.T) { activeDeadlineSeconds := int64(30) <ins> minID := int64(0) maxID := int64(2147483647) </ins> successCases := []api.PodSpec{ { // Populate basic fields, leave defaults for most. Volumes:       []api.Volume{{Name: "vol", VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{}}}},

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: "format": "int32" }, "revisionHistoryLimit": { <del> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 2.", </del> <ins> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.", </ins> "type": "integer", "format": "int32" },

All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps


Question: } } <ins> type fcInitSignal struct { nSignals int32 } func (s *fcInitSignal) SignalCount() int { return int(atomic.SwapInt32(&s.nSignals, 0)) } func (s *fcInitSignal) Signal() { atomic.AddInt32(&s.nSignals, 1) } func (s *fcInitSignal) Wait() { } type hookedListener struct { l        net.Listener onAccept func() } func (wl *hookedListener) Accept() (net.Conn, error) { wl.onAccept() return wl.l.Accept() } func (wl *hookedListener) Close() error { return wl.l.Close() } func (wl *hookedListener) Addr() net.Addr { return wl.l.Addr() } func TestFlowControlSignal(t *testing.T) { for _, tc := range []struct { Name           string Local          bool Available      bool Request        http.Request SignalExpected bool }{ { Name:           "local", Local:          true, SignalExpected: false, }, { Name:           "unavailable", Local:          false, Available:      false, SignalExpected: false, }, { Name:           "request performed", Local:          false, Available:      true, SignalExpected: true, }, { Name:      "upgrade request performed", Local:     false, Available: true, Request: http.Request{ Header: http.Header{"Connection": []string{"Upgrade"}}, }, SignalExpected: true, }, } { t.Run(tc.Name, func(t *testing.T) { okh := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) var sig fcInitSignal var signalCountOnAccept int32 backend := httptest.NewUnstartedServer(okh) backend.Listener = &hookedListener{ l: backend.Listener, onAccept: func() { atomic.StoreInt32(&signalCountOnAccept, int32(sig.SignalCount())) }, } backend.Start() defer backend.Close() p := proxyHandler{ localDelegate:   okh, serviceResolver: &mockedRouter{destinationHost: backend.Listener.Addr().String()}, } server := httptest.NewServer(contextHandler( http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { p.ServeHTTP(w, r.WithContext(utilflowcontrol.WithInitializationSignal(r.Context(), &sig))) }), &user.DefaultInfo{ Name:   "username", Groups: []string{"one", "two"}, }, )) defer server.Close() p.handlingInfo.Store(proxyHandlingInfo{ local:             tc.Local, serviceAvailable:  tc.Available, proxyRoundTripper: backend.Client().Transport, }) surl, err := url.Parse(server.URL) if err != nil { t.Fatalf("unexpected error: %v", err) } req := tc.Request req.URL = surl _, err = server.Client().Do(&req) if err != nil { t.Fatalf("unexpected error: %v", err) } if fired := (atomic.LoadInt32(&signalCountOnAccept) > 0); tc.SignalExpected && !fired { t.Errorf("flow control signal expected but not fired") } else if fired && !tc.SignalExpected { t.Errorf("flow control signal fired but not expected") } }) } } </ins> func getCertAndKeyPaths(t *testing.T) (string, string, string) { dir, err := ioutil.TempDir(os.TempDir(), "k8s-test-handler-proxy-cert") if err != nil {

On a cluster with many APIs provided by aggregated apiservers, with controllers managing those APIs, the number of occupied seats effectively has a floor of 1 per watch. If the combined max-inflight limit is reduced low enough, priority levels tend to become saturated by these long-running watch requests and other requests are starved. Watch requests consume a seat only during initialization. a PriorityLevelConfiguration and a corresponding FlowSchema to match watches against a resource served by an aggregated apiserver. some watch requests. the metric climb on the priority level per watch until all seats are exhausted. Kubernetes version (use ): (OpenShift 4.9) /sig api-machinery
cc
D'oh! I can see two obvious possible ways to cope: one apiserver not manage watch requests the are delegated to another apiserver; or a marker into the notification stream so that the non-origin apiservers can detect when initialization is done.
I would even say that for regular (non-long-running) requests, it's not obvious that they should be managed by P&F even. [I agree that for watch initialization it's particularly bad though.] - do we have a simple way to detect if the request will be served from our kube-apiserver or from some other aggregated apiserver?
There is not currently a way to know that ahead of time in the filters today. It require a list/watch, but it ought not be excessively expensive to make a 99% correct prediction, but the actual decision is made by the aggregated apiserver later in the request.
/triage accepted
: is there a way that we can plumb that knowledge forward to the filters?


Question: } var ( <del> shortDead int64                   = 10 longDead  int64                   = 1000000 noDead    int64                   = -12345 A         batch.ConcurrencyPolicy = batch.AllowConcurrent f         batch.ConcurrencyPolicy = batch.ForbidConcurrent R         batch.ConcurrencyPolicy = batch.ReplaceConcurrent T         bool                    = true F         bool                    = false </del> <ins> shortDead  int64                   = 10 mediumDead int64                   = 2 * 60 * 60 longDead   int64                   = 1000000 noDead     int64                   = -12345 A          batch.ConcurrencyPolicy = batch.AllowConcurrent f          batch.ConcurrencyPolicy = batch.ForbidConcurrent R          batch.ConcurrencyPolicy = batch.ReplaceConcurrent T          bool                    = true F          bool                    = false </ins> ) func TestSyncOne_RunOrNot(t *testing.T) { <ins> // Check expectations on deadline parameters if shortDead/60/60 >= 1 { t.Errorf("shortDead should be less than one hour") } if mediumDead/60/60 < 1 || mediumDead/60/60 >= 24 { t.Errorf("mediumDead should be between one hour and one day") } if longDead/60/60/24 < 10 { t.Errorf("longDead should be at least ten days") } </ins> testCases := map[string]struct { // sj spec concurrencyPolicy batch.ConcurrencyPolicy

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): , , Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: , deployed with Others: What happened: I am using a scheduledjob with . An instance of my job has taken a long time to finish, thus skipping many start times. Ever since, the scheduledjob will not schedule anything new. Controller manager log has: What you expected to happen: I expect the scheduledjob not to be stuck not scheduling anything anymore. With , I expect that it should just re-schedule something as soon as the previously running job has completed. The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues. ` How to reproduce it (as minimally and precisely as possible): The issue shows up after about 100 minutes of this running. Anything else do we need to know:
cc
sorry for the long delay in response, there were a few issues solved in later versions of ScheduledJob, can you verify this against latest 1.4.x? I've tried cherry-picking all of the crucial ones, like: and I'll try to verify this on my own as well.

I am still getting the same issue with
thanks for the info, will dig deeper into this one.
Thanks a lot any chance a potential fix could make it in 1.5?
if possible - yes, I'll try to backport it to 1.5
did you look into this? If not, let me know if you want me to pick it up and work on a PR
yeah, go ahead and ping me in the PR.
I just thought of something which might make sense. We've modified the error message to make it more "actionable", and the user can set a deadline to avoid running into this. However, the error message is still only controller level AFAIK and thus not exposed to the user. From the user's perspective, there is not way to tell why the CronJob is not scheduling the next instance. For these reasons, maybe we should also emit an event in the corresponding CronJob? I can send a small PR for this modification.


Question: return e } <ins> func podContainerFailed(c clientset.Interface, namespace, podName string, containerIndex int, reason string) wait.ConditionFunc { return func() (bool, error) { pod, err := c.CoreV1().Pods(namespace).Get(context.TODO(), podName, metav1.GetOptions{}) if err != nil { return false, err } switch pod.Status.Phase { case v1.PodPending: if len(pod.Status.ContainerStatuses) == 0 { return false, nil } containerStatus := pod.Status.ContainerStatuses[containerIndex] if containerStatus.State.Waiting != nil && containerStatus.State.Waiting.Reason == reason { return true, nil } return false, nil case v1.PodFailed, v1.PodRunning, v1.PodSucceeded: return false, fmt.Errorf("pod was expected to be pending, but it is in the state: %s", pod.Status.Phase) } return false, nil } } </ins> // LogPodStates logs basic info of provided pods for debugging. func LogPodStates(pods []v1.Pod) { // Find maximum widths for pod, node, and phase strings for column printing.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: N/A Which test(s) are failing: should fail substituting values in a volume subpath with absolute path Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Some of the tests are negative test cases which are supposed to ensure that those invalid usecases are handled properly. However, some of the tests are false positives, they can pass due to various reasons. One such example is: should fail substituting values in a volume subpath with absolute path. This test can pass if: the Pod cannot start due to various reasons (e.g.: the container image cannot be pulled or does not exist). the Pod ran to completion, even though the container was not supposed to start in the first place. Anything else we need to know: /sig testing Related:
/sig storage SIG storage owns the test case, though I see that you are changing core framework methods to fix this, so leaving SIG testing. Looks like also possibly sig-node, skimming the linked PRs Based on the PRs this also appears to be related to windows? (unsure, no mention here)
I've Related: to the issue description. After fixing this test, I've noticed that it now fails on Windows, which means the issue was masked by the test's false positive result. Basically, it means that you can have volume subpaths with absolute paths substituted on variable expansion and have Windows containers use them and be running, which according to the test, it shouldn't be a supported usecase. The fix should probably be also backported to previous releases.


Question: # Path for kube-proxy kubeconfig file on Windows nodes. export WINDOWS_KUBEPROXY_KUBECONFIG_FILE="${WINDOWS_K8S_DIR}kubeproxy.kubeconfig" # Pause container image for Windows container. <del> export WINDOWS_INFRA_CONTAINER="gcr.io/gke-release/pause-win:1.2.1" </del> <ins> export WINDOWS_INFRA_CONTAINER="gcr.io/gke-release/pause-win:1.5.1" </ins> # Storage Path for csi-proxy. csi-proxy only needs to be installed for Windows. export CSI_PROXY_STORAGE_PATH="https://storage.googleapis.com/gke-release/csi-proxy" # Version for csi-proxy

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add Support for Windows Server version 2004 and 20H2. Why is this needed: Windows Server version 1909 is going end of life.
/sig windows /triage accepted