Question: output_message=$(kubectl auth can-i get pods --subresource=log --quiet 2>&1 "${kube_flags[@]}"; echo $?) kube::test::if_has_string "${output_message}" '0' <ins> # kubectl auth can-i get '*' does not warn about namespaced scope or print an error output_message=$(kubectl auth can-i get '*' 2>&1 "${kube_flags[@]}") kube::test::if_has_not_string "${output_message}" "Warning" # kubectl auth can-i get foo does not print a namespaced warning message, and only prints a single lookup error output_message=$(kubectl auth can-i get foo 2>&1 "${kube_flags[@]}") kube::test::if_has_string "${output_message}" "Warning: the server doesn't have a resource type 'foo'" kube::test::if_has_not_string "${output_message}" "Warning: resource 'foo' is not namespace scoped" # kubectl auth can-i get pods does not print a namespaced warning message or a lookup error output_message=$(kubectl auth can-i get pods 2>&1 "${kube_flags[@]}") kube::test::if_has_not_string "${output_message}" "Warning" # kubectl auth can-i get nodes prints a namespaced warning message output_message=$(kubectl auth can-i get nodes 2>&1 "${kube_flags[@]}") kube::test::if_has_string "${output_message}" "Warning: resource 'nodes' is not namespace scoped" # kubectl auth can-i get nodes --all-namespaces does not print a namespaced warning message output_message=$(kubectl auth can-i get nodes --all-namespaces 2>&1 "${kube_flags[@]}") kube::test::if_has_not_string "${output_message}" "Warning: resource 'nodes' is not namespace scoped" </ins> fi # kubectl auth reconcile

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: if c.Started != nil && *c.Started { // Stop probing for startup once container has started. if w.probeType == startup { <del> return true </del> <ins> return false </ins> } } else { // Disable other probes until container has started.

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: func (r *ServiceRegistry) ExportService(ctx api.Context, name string, options unversioned.ExportOptions) (*api.Service, error) { r.mu.Lock() <del> defer r.mu.Lock() </del> <ins> defer r.mu.Unlock() </ins> return r.Service, r.Err }

Hi, I ran on Kubernetes at commit , filtered out benign issues and false positives and assembled the following report, grouped by category and sorted by severity in descending order. I hope this report is useful to you. The following line of code defers instead of : The following lines of code open files or connections and defer before checking the returned error. This will lead to panics in case of an error. In the following instances, values are being assigned to struct fields in methods with value receivers, without ever reading the fields again. This either suggests unnecessary assignments, or receivers that should be pointers instead. Per the testing.T documentation, T.FailNow, and by extension Fatal, Fatalf and so on, must be called on the same goroutine as the test, not in other goroutines. That invariant is violated in the following places: The following instances use Printf even though they don't need format strings. In some cases, the inputs are user provided, which can lead to incorrect output: The following condition compares with itself. That's either pointless, or a possible copy & paste eror: These values of are never evaluated: The following loops always loop at most once, indicating bugs or unnecessary control flow constructs: The following arguments are being overwritten before their first use. Judging from context, this is either because of missing pointer dereferences, or because of useless arguments: The same error class as assigned but not read errors, but indicating left over useless code, not missing error checking: All of these slices are never used. Most are lists of things that are discarded, some are values that are subsequently overwritten unconditionally.
Opened for the aws codesmell issue. Could we get staticcheck into e2e I wonder?
/assign Is this still an issue?
/unassign
You could run on a recent commit of Kubernetes to confirm whether the reported issues have already been resolved.
someone pay this guy, that was incredible.
We should scrub this and run it as a verify script.
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/lifecycle frozen /remove-lifecycle stale
/remove-lifecycle rotten
/cc
I started opening a bunch of PRs for this:
We do run staticcheck now and we've had various PRs regarding fixing code bugs and removing them from the failures list, but there are still many failing packages ... I think fixing the failures should be a distinct bug from adding staticcheck to CI (done in ) and is mostly what has been discussed here after the initial comment. I've filed a "good first" issue with more & updated detail for newcomers to help fix these. Thanks everyone who's helped so far and for reporting. /remove-sig contributor-experience /close
Closing this issue. <detailsIn response to : We do run staticcheck now and we've had various PRs regarding fixing code bugs and removing them from the failures list, but there are still many failing packages ... I've filed a "good first" issue with more & updated detail for newcomers to help fix these. /remove-sig contributor-experience /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: mapper, typer := f.Object() // TODO: use resource.Builder instead <del> obj, err := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand(cmd)). </del> <ins> obj, err := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand()). </ins> NamespaceParam(cmdNamespace).RequireNamespace(). FilenameParam(filename). Do().

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: cat "${file}" echo <del> echo "= ${file} $(md5sum < "${file}" | awk '{print $1}')" </del> <ins> echo "= ${file} $(kube::util::md5 "${file}")" </ins> echo "================================================================================" echo done >> ${TMP_LICENSE_FILE}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: run on macOS, all md5 hashes will be stripped from the output and the script will not fail or warn. What you expected to happen: The script should work or detect this issue and fail. Currently it silently ignores not existing on the host. We have some warning logic, but solely based on bash version, if the user upgraded their bash installation but did not install md5sum we don't warn them. How to reproduce it (as minimally and precisely as possible): run the script Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig testing /priority important-longterm /assign
cc


Question: func TestModifyHostConfig(t *testing.T) { setNetworkHC := &dockercontainer.HostConfig{} <ins> // When we have Privileged pods, we do not need to use the // Masked / Readonly paths. </ins> setPrivSC := &runtimeapi.LinuxContainerSecurityContext{} setPrivSC.Privileged = true <ins> setPrivSC.MaskedPaths = []string{"/hello/world/masked"} setPrivSC.ReadonlyPaths = []string{"/hello/world/readonly"} </ins> setPrivHC := &dockercontainer.HostConfig{ Privileged: true, } <ins> unsetPrivSC := &runtimeapi.LinuxContainerSecurityContext{} unsetPrivSC.Privileged = false unsetPrivSC.MaskedPaths = []string{"/hello/world/masked"} unsetPrivSC.ReadonlyPaths = []string{"/hello/world/readonly"} unsetPrivHC := &dockercontainer.HostConfig{ Privileged:    false, MaskedPaths:   []string{"/hello/world/masked"}, ReadonlyPaths: []string{"/hello/world/readonly"}, } </ins> setCapsHC := &dockercontainer.HostConfig{ CapAdd:  []string{"addCapA", "addCapB"}, CapDrop: []string{"dropCapA", "dropCapB"},

Is this a BUG REPORT or FEATURE REQUEST?: kind bug What happened: After execute kubeadm init during kube-proxy pod initialization there is CrashLoopBackOff error all the time Logs from pod: [centos ~]$ kubectl logs kube-proxy-8gs45 -n kube-system What you expected to happen: kube-proxy pod run properly How to reproduce it (as minimally and precisely as possible): follow instruction: try to execute: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
/sig network
I have exactly the same problem with the 1.12.0-beta.1. Does kube-proxy really need access to /proc/sys/* in this case?
Same problem with Ubuntu
Does that mean kube-proxy can only be run outside of container?
/sig cluster-lifecycle
/milestone v1.12
Looks like docker inspect shows masked and read-only paths (looking at ) even when the pod is
It being close to release, since this is not urgent, please save this for 1.13! /milestone clear /milestone v1.13
/milestone v1.13
/priority critical-urgent /milestone v1.12 Sorry the activity and urgency is reflected in the PR :)
welp! sorry about that!
I'm still seeing this on RC1, did this land before the release was cut? /cc
update from :
I just verified on release-1.12 branch and it's good.


Question: - file: /etc/init.d/kubelet {% endif %} - file: /var/lib/kubelet/kubernetes_auth <del> {% if grains.network_mode is defined and grains.network_mode == 'openvswitch' %} - sls: sdn {% endif %} </del>

It appears that the provision-network script that is run on the kubernetes-master can cause salt provisioning on the minions to fail or block in a forever retry timeout error loop. I think it's because of the service network.restart that happens after openvswitch is installed in provision-network. I only say this because when I disable installing docker on kubernetes-master in vagrant, I do not have an issue. - is there a different way we can structure this on the kubernetes-master? For example, could we run the script in or something?


Question: "type": "string" }, "healthCheckNodePort": { <del> "description": "healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type).", </del> <ins> "description": "healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type). This field cannot be updated once set.", </ins> "format": "int32", "type": "integer" },

We are currently using v1.19 and it seems that if you have a service with set to then changing the health check node port is not supported. One way to get around this is by changing the to and then changing it back to in addition to specifying the with the desired port number. This behavior is not very well documented and the below error message is not very useful: I wonder if we could also add another case below to release the current health check port and allow changing it to the user specified value given it's valid? It's possible that after creating a someone wants to change it to a user-provided and the current way to do it is only by toggling the . This feature with the documentation would be more straightforward way to make any port changes.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
19 is EOL, last supported version is 1.22 It seems the message has changed since then, this area had important changes in latest releases that improved the API behavior However, the field still can not be updated once set, what you do switching the is wiping the value and reallocating it. It is a documentation problem then
IMO, the amount of effort it would take to make this mutable is not justified when there is a workaround. Convince me the workaround doesn't work? Overall HCNP is a buried implementation detail.
I can't speak much to the amount of effort, but I can speak a bit to the value: a user, I don't expect healthcheckNodePort to be immutable. I'm not sure if my mental model is too simplistic, but it feels like a part of the configuration I control, and I can change NodePorts, so why not the healthcheckNodePort. the work around isn't necessarily obvious. It also takes experimenting with to know that it'll even work. this in a production environment is not trivial.

If changing the healthcheckNodePort was supported then you could change the config to the state you want it to be, and your rollout orchestration tooling goes and applies it to your collection of k8s clusters. With the work around you now have to have a 2 step process. After step 1 of this process your service is using a traffic policy which might break your application (say you depend on client ip). So to avoid the downtime you'd have to build some reasonably off-pattern orchestration that doesn't just apply the configuration from the commit you used for your release, but instead two commits, or two configs, etc. Operationally this isn't actually that cheap to build, especially as this is likely to be a one-off operation and not something you do every release. Not exactly the same thing, but related to healthcheckNodePort, I've also found that specifying it explicitly vs not explicitly in the config, and switching between the two modes, has unexpected behavior as well. If you started out as having it specified, applying a config with it unspecified tries to allocate a new value, and complains that it can't change healthcheckNodePort for traffic policy local. Why not just use the already allocated value? Going from unallocated to specifying it explicitly at the same value seems to works fine.
The latter issue should be fixed in recent releases - what version are you seeing that on? I understand it would be more consistent to be mutable but there are real downstream implications to changing it, code complexity, test complexity, etc, and I don't understand why you want to change it in the first place. Humans should not be paying attention to that field. It should not even be in spec, but for historical reasons.
Same version as original post, v1.19. Glad to hear that issue is fixed in later versions! We're making steady progress updating, but not all caught up yet. I don't want to insist on this fix, as I said I'm not aware of the fix complexity and leave it up to you guys to balance that out, just wanted to provide the value perspective. As for not being exposed to users at all, I think we'd probably have been better off if we never knew about it.

Unfortunately our original configs had it explicitly set, eventually ran into conflicts in various k8s clusters and tried to transition to having it be unspecified, and thus ran into the above situation.
Yeah, there's really no good reason to set it, pretty much ever, and lots of reasons not to. I will cross-check that it is fixed if I get a moment tomorrow.
It has been fixed by you Tim
I thought so, I just wanted to triple check :)


Question: t.Errorf("expected success or bad request err, got %#v", err) } }) <ins> t.Run("ApplyPatchType should handle a valid patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"apiVersion":"v1","kind":"Secret"` + strings.Repeat(" ", 3*1024*1024-100) + `}`) err = rest.Patch(types.ApplyPatchType).Param("fieldManager", "test").AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")). Body(patchBody).Do().Error() if err != nil { t.Errorf("unexpected error: %v", err) } }) </ins> t.Run("Delete should limit the request body size", func(t *testing.T) { err = c.Delete().AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")). Body(hugeData).Do().Error()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: onExecuteFunc := func() { firstRunning.Done() <del> firstRunning.Wait() sendSignals() </del> fakeFilter.wait() allRunning.Done()

Testgrid doesn't load kubernetes/test-infra From: /sig api-machinery
/assign /triage accepted
Yes. It looks like a bug. There are many calls to , and they are not all guaranteed to happen before the second . I will revise.


Question: "strings" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/json" </ins> "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/resource" "k8s.io/client-go/rest/fake"

Currently, the cmd output msg is not distinguished between label and unlabel (using suffix ), both of them response . To distinguish between label and unlabel msg: , just like .
/sig cli
/triage accepted


Question: listerWatcher ListerWatcher // backoff manages backoff of ListWatch backoffManager wait.BackoffManager <del> // initConnBackoffManager manages backoff the initial connection with the Watch call of ListAndWatch. initConnBackoffManager wait.BackoffManager resyncPeriod           time.Duration </del> <ins> resyncPeriod   time.Duration </ins> // clock allows tests to manipulate time clock clock.Clock // paginatedResult defines whether pagination should be forced for list calls.

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: } }) <del> testName = "should survive network partition." </del> <ins> // TODO: this test has nothing to do with resizing nodes so it should be moved elsewhere. // TODO: two things are tested here: // 1. pods from a uncontactable nodes are rescheduled // 2. when a node joins the cluster, it can host new pods. // Factor out the cases into two separate tests. testName = "Uncontactable nodes, have their pods recreated by a replication controller, and can host new pods after rejoining." </ins> It(testName, func() { if testContext.CloudConfig.NumNodes < 2 { By(fmt.Sprintf("skipping %s test, which requires at least 2 nodes (not %d)",

Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of


Question: return nil, err } <ins> emptyDuration := MyDuration{} if cfg.Metadata.RequestTimeout == emptyDuration { cfg.Metadata.RequestTimeout.Duration = time.Duration(defaultTimeOut) } provider.HTTPClient.Timeout = cfg.Metadata.RequestTimeout.Duration </ins> os := OpenStack{ provider:     provider, region:       cfg.Global.Region,

What happened: We use OpenStack as our cloud provider. We meet an issue that some controllers stuck because all workers routine stuck at http call to OpenStack services because of rabbitmq issues. Stuck routine call stack: Checked the openstack cloud provider that there is no HttpClient.Timeout set which means no timeout during http call. I think we should set at least the default http timeout value. Environment: Kubernetes version (use ): v1.6.3 Cloud provider or hardware configuration**: OpenStack
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /kind feature PTAL PR .


Question: "//pkg/kubelet/apis/kubeletconfig:go_default_library", "//pkg/kubelet/checkpointmanager:go_default_library", "//pkg/kubelet/checkpointmanager/checksum:go_default_library", <ins> "//pkg/kubelet/checkpointmanager/errors:go_default_library", </ins> "//pkg/kubelet/cm:go_default_library", "//pkg/kubelet/container:go_default_library", "//pkg/kubelet/dockershim/cm:go_default_library",

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: aggrclient = f.AggregatorClient }) <del> It("Should be able to support the 1.7 Sample API Server using the current Aggregator", func() { </del> <ins> It("Should be able to support the 1.10 Sample API Server using the current Aggregator", func() { </ins> // Make sure the relevant provider supports Agggregator framework.SkipUnlessServerVersionGTE(serverAggregatorVersion, f.ClientSet.Discovery()) framework.SkipUnlessProviderIs("gce", "gke") <del> // Testing a 1.7 version of the sample-apiserver </del> <ins> // Testing a 1.10 version of the sample-apiserver </ins> TestSampleAPIServer(f, imageutils.GetE2EImage(imageutils.APIServer)) }) })

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: "${NODE_INSTANCE_PREFIX}-group") if [[ "$deleteCmdOutput" != ""  ]]; then # Managed instance group deletion is done asyncronously, we must wait for it to complete, or subsequent steps fail <del> deleteCmdOperationId=$(echo $deleteCmdOutput | grep "Operation:" | sed "s/.*Operation:s//" | sed "s/s.*//" | sed "s/ //g") </del> <ins> deleteCmdOperationId=$(echo $deleteCmdOutput | grep "Operation:" | sed "s/.*Operation:[[:space:]]*([^[:space:]]*).*/1/g") </ins> if [[ "$deleteCmdOperationId" != ""  ]]; then deleteCmdStatus="PENDING" while [[ "$deleteCmdStatus" != "DONE" ]] do <del> sleep 5 </del> <ins> sleep 5 </ins> deleteCmdOperationOutput=$(gcloud preview managed-instance-groups --zone "${ZONE}" get-operation $deleteCmdOperationId) <del> deleteCmdStatus=$(echo $deleteCmdOperationOutput | grep -i "status:" | sed "s/.*status:s//" | sed "s/s.*//" | sed "s/ //g") </del> <ins> deleteCmdStatus=$(echo $deleteCmdOperationOutput | grep -i "status:" | sed "s/.*status:[[:space:]]*([^[:space:]]*).*/1/g") </ins> echo "Waiting for MIG deletion to complete. Current status: " $deleteCmdStatus done fi

is running in to an issue where the operation ID pulled from the MIG delete command during GCE kube-down is incorrect:
Looks like this is caused by OS X using non-GNU . A couple of different possible fixes, one is to force check for and use that when available. Another approach, courtesy of is to just limit our self to non-GNU .


Question: {HostPort: 443, ContainerPort: 444}, }, }, <ins> { "", []api.Port{}, }, </ins> } for _, tt := range testCases { ports := portsFromString(tt.spec)

$ run thockin/hostnametcp 1 hostname-tcp E1002 14:48:23. ] Container port is not integer: strconv.ParseInt: parsing "": invalid syntax


Question: return nil } <ins> func makeEnvVariableName(str string) string { return strings.ToUpper(strings.Replace(str, "-", "_", -1)) } </ins> func makeLinkVariables(service api.Service, machine string) []api.EnvVar { <del> prefix := strings.ToUpper(service.ID) </del> <ins> prefix := makeEnvVariableName(service.ID) </ins> var port string if service.ContainerPort.Kind == util.IntstrString { port = service.ContainerPort.StrVal } else { port = strconv.Itoa(service.ContainerPort.IntVal) } <del> portPrefix := prefix + "_PORT_" + strings.ToUpper(strings.Replace(port, "-", "_", -1)) + "_TCP" </del> <ins> portPrefix := prefix + "_PORT_" + makeEnvVariableName(port) + "_TCP" </ins> return []api.EnvVar{ { Name:  prefix + "_PORT",

In issue it appeared that consensus was reached that a service id could contain a hyphen, but that it would be translated to an underscore in the environment variables (this issues was the root problem, back in ). However, the replacement does not appear to have been done: hyphens are still not replaced with underscores in env-var names. (Although now underscores are blocked in service names!)
Hm, that's weird. I thought that this code was doing replacement. Did that get removed?
Thanks for looking into this lavalamp. Was your question directed at me? I don't see any string replacement code in the code you linked to, at least not for the (I do see escaping for the port in line 198)
Ah, no, I was just thinking out loud there. Maybe knows what happened?
Ah, that's an oversight. We do it for the Docker link style variables, but not the old school ones. I'll send a PR.
PR sent.


Question: visibility = ["//visibility:public"], deps = [ "//pkg/api/v1/pod:go_default_library", <ins> "//pkg/apis/core/validation:go_default_library", </ins> "//pkg/scheduler/algorithmprovider:go_default_library", "//pkg/scheduler/apis/config:go_default_library", "//pkg/scheduler/apis/config/scheme:go_default_library",

What happened: When a pod can't be scheduled and the cardinality of taints on nodes is high, the Note field in the event can become bigger than the 1024 allowed and the event isn't sent. The pod stays in a state of without en event which makes the User Experience very bad What you expected to happen: I would expect the Kube-scheduler to be able to send the event no matter how many different taints are on nodes How to reproduce it (as minimally and precisely as possible): add many different taints to nodes and then trying to schedule something which can't be successfully scheduled (higher CPU/memory than any node has) Anything else we need to know?: In this case the length of the message was 1334 characters. Environment: Kubernetes version (use ): 1.18.9 Cloud provider or hardware configuration: bare metal Possible solutions: make event size configurable, it's now hardcoded to 1024 characters make scheduler note aware of that limit and work around I, if length 1024 then truncate the string (so at least something is shown) produce N events splitting the text (preferred one): change event Note to be to something like: (the whole complete information would still be present in the pod status field. for debugging purposes)
/sig scheduling
/triage accepted /cc
I could provide the fix in case we define which solution is best
This is a consequence of I think we can just drop some reasons when we are building the status message if the message is too large. This comes from here /help
/remove-help /assign
how many different taints did you have? I think in all cases we should truncate the message as well:
Another remidiation is to supress outoutping nil fields (such as GenerateName:"", SelfLink:"", UID:"", ResourceVersion:"", etc.) in the EventRecorder side.
The size limit we're hitting is specifically on the field
Completely agreed
good issue~
/assign
/unassign


Question: "testing" "time" <del> cadvisorapi "github.com/google/cadvisor/info/v1" </del> "io/ioutil" <ins> "os" cadvisorapi "github.com/google/cadvisor/info/v1" </ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state" <ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology" </ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" <del> "os" </del> ) type mockState struct {

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work?

I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!

Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs?

If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?


Question: return fmt.Errorf("pod %s on node %s had incorrect containers: %#v", pod.Name, pod.Spec.NodeName, pod.Status.ContainerStatuses) } <del> if status := findContainerStatusInPod(pod, "blocked"); status != nil { </del> <ins> if status := e2epod.FindContainerStatusInPod(pod, "blocked"); status != nil { </ins> if (status.Started != nil && *status.Started == true) || status.LastTerminationState.Terminated != nil || status.State.Waiting == nil { return fmt.Errorf("pod %s on node %s should not have started the blocked container: %#v", pod.Name, pod.Spec.NodeName, status) } } <del> status := findContainerStatusInPod(pod, "fail") </del> <ins> status := e2epod.FindContainerStatusInPod(pod, "fail") </ins> if status == nil { return fmt.Errorf("pod %s on node %s had incorrect containers: %#v", pod.Name, pod.Spec.NodeName, pod.Status) }

We would like to know early if the behavior is broken on any container runtime implementation. This is motivated by the discussion under the PR:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // applySandboxLinuxOptions applies LinuxPodSandboxConfig to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxLinuxOptions(hc *dockercontainer.HostConfig, lc *runtimeapi.LinuxPodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string) error { // Apply Cgroup options. <del> // TODO: Check if this works with per-pod cgroups. hc.CgroupParent = lc.GetCgroupParent() </del> <ins> cgroupParent, err := ds.GenerateExpectedCgroupParent(lc.GetCgroupParent()) if err != nil { return err } hc.CgroupParent = cgroupParent </ins> // Apply security context. applySandboxSecurityContext(lc, createConfig.Config, hc, ds.networkPlugin)

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: type priorityMetadata struct { podLimits               *schedulernodeinfo.Resource podSelector             labels.Selector <del> controllerRef           *metav1.OwnerReference </del> podFirstServiceSelector labels.Selector podTopologySpreadMap    *podTopologySpreadMap }

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: o.Encoder = f.JSONEncoder() o.ShortOutput = cmdutil.GetFlagString(cmd, "output") == "name" o.Record = cmdutil.GetRecordFlag(cmd) <ins> o.Local = cmdutil.GetFlagBool(cmd, "local") </ins> o.ChangeCause = f.Command(cmd, false) o.PrintObject = f.PrintObject o.Cmd = cmd

It tries to contact the api server for the discovery docs and fails because no api server is running. Could first look in its local cache if it's passed --local?
I think here has similar meaning as , it will not contact api-server to do real patch, but it will fetch the resource from api-server. What you mean look in it's local cache? to find the resource in local cache? I think you have specified the resource in your command line.
Use of --local flag should completely eliminate communication with api-server I have sent a PR which should fix this issue A command like kubectl set image --local -f busybox=busybox:1.26 -o yaml should load the local yaml file and print out the patched version (image should get updated to busybox:1.26) in yaml format on the screen. With the submitted patch such behavior is achived


Question: r.backoff.UpdateBackoff(r.URL(), err, resp.StatusCode) } if err != nil { <del> // "Connection reset by peer", "Connection refused" or "apiserver is shutting down" are usually a transient errors. </del> <ins> // "Connection reset by peer" or "apiserver is shutting down" are usually a transient errors. </ins> // Thus in case of "GET" operations, we simply retry it. // We are not automatically retrying "write" operations, as // they are not idempotent.

/kind bug changed behavior of client-go on connection refused errors, which changed a short wait and an "invalid host/port" error into a long retry wait and a confusing "internal server" error /assign Originally posted by in
/sig api-machinery
/milestone v1.18
I linked
Hello, bug-triage here. Just want to make sure we are still staffed and on-track for v1.18 code freeze
thanks, I will get to this tomorrow.
gentle ping


Question: EnvironmentFile=-/run/flannel/docker EnvironmentFile=-/opt/kubernetes/cfg/docker WorkingDirectory=/opt/kubernetes/bin <del> ExecStart=/opt/kubernetes/bin/docker daemon $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPTS </del> <ins> ExecStart=/opt/kubernetes/bin/dockerd $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPTS </ins> LimitNOFILE=1048576 LimitNPROC=1048576

Did a git clone and trying to build on centos. [root kubernetes]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root kubernetes]# make release +++ [0616 17:42:23] Verifying Prerequisites.... +++ [0616 17:42:57] Building Docker image kube-build:build-. +++ [0616 17:44:44] Running build command.... Usage of loopback devices is strongly discouraged for production use. Either use or use to suppress this warning. Go version: go version go1.6.2 linux/amd64 +++ [0616 17:45:01] Multiple platforms requested, but available 0G < threshold 11G, building platforms in serial +++ [0616 17:45:01] Building the toolchain targets: +++ [0616 17:45:04] Building go targets for linux/amd64: cmd/kube-dns cmd/kube-proxy cmd/kube-apiserver cmd/kube-controller-manager cmd/kubelet cmd/kubemark cmd/hyperkube federation/cmd/federation-apiserver federation/cmd/federation-controller-manager plugin/cmd/kube-scheduler go build /usr/local/go/pkg/tool/linuxamd64/compile: signal: killed !!! Error in 'go install "{goflags[ -ldflags "{nonstatics[ exited with status 1 Call stack: 1: kube::golang::buildbinariesforplatform(...) 2: kube::golang::buildbinaries(...) 3: hack/build- main(...) Exiting with status 1 !!! Error in '( kube::golang::setupenv; echo "Go version: (kube::golang::hostplatform); local goflags goldflags; eval "goflags=({KUBEGOLDFLAGS:-} do if [[ "${arg}" == "--usegobuild" ]]; then usegobuild=true; else if [[ "${arg}" == - ]]; then goflags+=("${arg}"); else targets+=("${arg}"); fi; fi; done; if [[ ${#targets[ -eq 0 ]]; then targets=("${KUBEALLTARGETS[ fi; local -a platforms=("{KUBEBUILDPLATFORMS[ if [[ ${#platforms[ -eq 0 ]]; then platforms=("${hostplatform}"); fi; local binaries;

binaries=({targets[ local parallel=false; if [[ ${#platforms[ -gt 1 ]]; then local gigs; gigs={gigs} -ge ${KUBEPARALLELBUILDMEMORY} ]]; then kube::log::status "Multiple platforms requested and available {KUBEPARALLELBUILDMEMORY}G, building platforms in parallel"; parallel=true; else kube::log::status "Multiple platforms requested, but available {KUBEPARALLELBUILDMEMORY}G, building platforms in serial"; parallel=false; fi; fi; kube::golang::buildkubetoolchain; if [[ "${parallel}" == "true" ]]; then kube::log::status "Building go targets for {targets[ local platform; for platform in "${platforms[ do ( kube::golang::setplatformenvs "{platform}: go build started"; kube::golang::buildbinariesforplatform {usegobuild:-}; kube::log::status "{platform////}.build" & done; local fails=0; for job in $(jobs -p); do wait ${job} let "fails+=1"; done; for platform in "${platforms[ do cat "/tmp//${platform////}.build"; done; exit ${fails}; else for platform in "${platforms[ do kube::log::status "Building go targets for {targets[ kube::golang::setplatformenvs "{platform} ${usegobuild:-}; done; fi )' exited with status 1 Call stack: 1: kube::golang::buildbinaries(.) 2: hack/build- main(.) Exiting with status 1 !!! Error in '" exited with status 1 Call stack: 1: kube::build::runbuildcommand(...) 2: main(...) Exiting with status 1 make:  [release] Error 1
Note that Centos downloads a bad "docker" binary that is an image with "404 Error". Centos seems abandonned by Kubernetes team... so I needed to make the entire operation with Ansible. And that is not simple at all :)
There was below info from your log: It seems like that the compiler was OOM killed due to memory pressure. AFAIK, consumes a lot of memory. I used to encounter when running it. We're going to close this issue.

Feel free to file a new issue if you fail to run at latest HEAD and you think there is a bug. Thanks!


Question: // We used more than allowed. return } <ins> // add the unused time directly to the budget // takeAvailable() will take into account the elapsed time </ins> if t.budget = t.budget + unused; t.budget > t.maxBudget { t.budget = t.maxBudget }

Instead of relying on the clock: we should do something similar to the golang rate-limitter is doing:
/sig api-machinery /priority important-longterm
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to help on this.
/assign Thanks! /triage accepted
can you please provide more context or some guidance on how to proceed?
Sorry for delayed response. - what we should do is to basically: instead of having an additional go-routine that every 1s refreshes the budget change it roughly to: (a) keep the value of budget and the last time when we were refreshing it (b) on takeAvailable method: first recompute takeAvailable, which will be "min(current budget + time elapsed since last refresh * refreshValue, maxAllowedBudget) update last refresh time to now return that value [We also would need to update the existing tests to match the new logic.] - would you be willing to work on it?
-- I can take this up. One question I had was should the method rely on or the current time should be passed on by the caller like in the following: Having the caller pass the time completely removes the dependency on the clock at least for the implementation.
/assign
Let's may be start with: passing on budget creation to it relying on
-- thank you for the thoughts. I drafted a rough implementation in which still depends on . I will make the change as you suggested and rely solely on a instance passed on initialization. I have to rethink the testing strategy a bit, possibly using a .
Yes - FakeClock is the way to test this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopening - this didn't happen.


Question: "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/validation" "k8s.io/apimachinery/pkg/util/validation/field" <del> utilfeature "k8s.io/apiserver/pkg/util/feature" </del> "k8s.io/kubernetes/pkg/apis/apps" api "k8s.io/kubernetes/pkg/apis/core" apivalidation "k8s.io/kubernetes/pkg/apis/core/validation" <del> "k8s.io/kubernetes/pkg/features" </del> ) // ValidateStatefulSetName can be used to check whether the given StatefulSet name is valid.

This applies to StatefulSetStartOrdinal feature gate. Field validation should be done regardless of the feature gate state. From
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig apps


Question: framework.Logf("Pod %v succeeded ", pod.Name) return nil } <ins> func conditionType(condition interface{}) string { return string(condition.(v1.PersistentVolumeClaimCondition).Type) } </ins>

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } _, exists, err := e.store.Get(namespace) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } if exists { return nil

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: } // Hostpath <del> type hostpathDriver struct { </del> <ins> type hostPathDriver struct { </ins> node v1.Node driverInfo DriverInfo } <del> var _ TestDriver = &hostpathDriver{} var _ PreprovisionedVolumeTestDriver = &hostpathDriver{} var _ InlineVolumeTestDriver = &hostpathDriver{} </del> <ins> var _ TestDriver = &hostPathDriver{} var _ PreprovisionedVolumeTestDriver = &hostPathDriver{} var _ InlineVolumeTestDriver = &hostPathDriver{} </ins> <del> // InitHostpathDriver returns hostpathDriver that implements TestDriver interface func InitHostpathDriver() TestDriver { return &hostpathDriver{ </del> <ins> // InitHostpathDriver returns hostPathDriver that implements TestDriver interface func InitHostPathDriver() TestDriver { return &hostPathDriver{ </ins> driverInfo: DriverInfo{ <del> Name:        "hostpath", </del> <ins> Name:        "hostPath", </ins> MaxFileSize: testpatterns.FileSizeMedium, SupportedFsType: sets.NewString( "", // Default fsType

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: var ( explainLong = templates.LongDesc(i18n.T(` <del> List the fields for supported resources. </del> <ins> Describe fields and structure of various resources. </ins> This command describes the fields associated with each supported API resource. Fields are identified via a simple JSONPath identifier: <type>.<fieldName>[.<fieldName>] <del> Add the --recursive flag to display all of the fields at once without descriptions. </del> Information about each field is retrieved from the server in OpenAPI format.`)) explainExamples = templates.Examples(i18n.T(` # Get the documentation of the resource and its fields kubectl explain pods <ins> # Get all the fields in the resource kubectl explain pods --recursive # Get the explanation for deployment in supported api versions kubectl explain deployments --api-version=apps/v1 </ins> # Get the documentation of a specific field of a resource <del> kubectl explain pods.spec.containers`)) </del> <ins> kubectl explain pods.spec.containers # Get the documentation of resources in different format kubectl explain deployment --output=plaintext-openapiv2`)) </ins> plaintextTemplateName          = "plaintext" plaintextOpenAPIV2TemplateName = "plaintext-openapiv2"

Especially the options description are super sad. I think we can improve them in the next beta iteration cc /sig cli
/triage accepted
Hi does fixes this issue?
I agree with the same. Unlike other outputs, it doesn't share the details about the values or their usage.
Unfortunately, I don't think it does. Have you tried it? Please send a pull-request I'll happily review it.
Hi , I'd like to work on this. However, I would need some guidance so I know I'm in the right direction. My guess is we change the text I have noticed these descriptions are not as per the other flag descriptions in kubectl. I'll make a draft once you confirm.
For eg these are options in : we can try adding details and formal tone to the text. /assign
I don't know if you need my help, you seem to be doing just fine. Please go ahead and ask me questions if you have specific problems, I'll be happy to guide. Thanks!
That's correct.
sure thing , I'll send the PR
Hi How does this output looks :
Hi Raised the PR: for reviews on same.


Question: requests: memory: 20Mi cpu: 10m <ins> securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 1001 </ins> dnsPolicy: Default  # Don't use cluster DNS. serviceAccountName: kube-dns

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Apply security best practices for both kube-dns and core-dns. Low hanging fruit: [x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities[x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities This is based on the long term issue and implemented by PR . Why is this needed: Our system pods should run using security best practices, both to enhance cluster security and serve as examples of best practices to users.
/sig auth /area security


Question: Log-Output ("Skip: Fluentbit logging agent is already installed") return } DownloadAndInstall-LoggingAgents Create-LoggingAgentServices }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There's a race condition between starting Windows service and when the VMs network is actually available to communicate with the VM's metadata server. If the service starts first and cannot communicate with the metadata server or start talking to the stackdriver logging service it'll crash and not resume because the service is not configured to restart on failure. Typically a is used for a service like this but Windows does not support privileged containers so a standard Windows Service is used. What you expected to happen: If the service fails to start it should be automatically be restarted. How to reproduce it (as minimally and precisely as possible): This is a flaky issue but a way to repro this is to effectively by RDP'ing into the node and running: Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 and onward Cloud provider or hardware configuration: GCE OS (e.g: ): Windows Kernel (e.g. ): Windows (any version) Install tools: cluster/gce/windows/README-GCE-Windows-kube- Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign jeremyedwards
GitHub didn't allow me to assign the following users: jeremyedwards. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign jeremyedwards Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: readOnly: true nodeSelector: beta.kubernetes.io/fluentd-ds-ready: "true" <del> tolerations: - key : "node.alpha.kubernetes.io/ismaster" effect: "NoSchedule" </del> terminationGracePeriodSeconds: 30 volumes: - name: varlog

Fluentd-elastcisearch sends logs to an Elasticsearch installation inside the cluster. It uses ES service to access, so it doesn't work on master, because there's no kube-proxy on master. Because of that, introducing master toleration to in was unnecessary.


Question: "github.com/prometheus/client_golang/prometheus" corev1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/klog" "k8s.io/kubernetes/pkg/features"

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: } } <del> func getNodeWithPodAndVolumeLimits(pods []*v1.Pod, limit int64, filter string) *schedulernodeinfo.NodeInfo { </del> <ins> func getNodeWithPodAndVolumeLimits(pods []*v1.Pod, limit int64, driverNames ...string) *schedulernodeinfo.NodeInfo { </ins> nodeInfo := schedulernodeinfo.NewNodeInfo(pods...) node := &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "node-for-max-pd-test-1"}, Status: v1.NodeStatus{ <del> Allocatable: v1.ResourceList{ getVolumeLimitKey(filter): *resource.NewQuantity(limit, resource.DecimalSI), }, </del> <ins> Allocatable: v1.ResourceList{}, </ins> }, } <ins> for _, driver := range driverNames { node.Status.Allocatable[getVolumeLimitKey(driver)] = *resource.NewQuantity(limit, resource.DecimalSI) } </ins> nodeInfo.SetNode(node) return nodeInfo }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Volume binding mode doesn't consider the volume attach limit per node. What you expected to happen: volume binding mode should have respected the attach limit, similar to binding mode. How to reproduce it (as minimally and precisely as possible): In CSI driver, configured the volume-attach limit to 1 per node on a 3 node cluster. In volume binding mode, only 3 pods with 3 PV/PVCs runs. In binding mode, all 5 pods runs with 5PV/PVCs on 3 node cluster. (Actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled) Anything else we need to know?: I was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 Volume limit and 500 pods running on the 5 mode cluster. Environment: Kubernetes version (use ): ~ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On-prem linux server OS (e.g. from /etc/os-release): Centos 7Kernel (e.g. ): 3.10.0 Install tools: Others:
cc
/sig storage
/assign Currently the predicate PVCs that are not bound. It should lookup the storageclass to get the provisioner/driver name. This solution will require a StorageClass object is even if you don't use dynamic provisioning.

I had a thought that it wouldn't be so simple to count unbound PVCs of other pods since binding is still in progress, but I think it's fine actually. We only count other pods that have been assigned to the same node. Those pods are assigned to the node AND still unbound because of one of these reasons: It's actually been scheduled to the node and someone manually came in and unbound the PVC: unlikely and an unsupported sequence since we implemented StorageProtection The Pod was initially created with nodeName already set, bypassing the scheduler The Pod is going to be scheduled (it's been assumed), and it's currently going through binding. This is normal with delayed binding. Either way, the fact that the Pod's nodename is set to this node means that the pod should be counting towards the node's limit.
Hmm, so after opening the PR that fixes this, I realize that - from storageclass we don't really know if the volume we are evaluating is CSI volume or not. I think a similar problem exists for existing predicate though. Like AzureDisk predicate does not know if PVC it is over counting is of type AzureVolume or not. So, I suspect that those 4 hardcoded predicates are counting all delayed binding volumes.
if the storageclass.provisioner name is not found, can we just ignore it?
I think we can ignore it. The PR currently does that.


Question: falseBytes = []byte("false") ) <del> func toUnstructured(sv, dv reflect.Value) error { st, dt := sv.Type(), dv.Type() </del> <ins> func getMarshaler(v reflect.Value) (encodingjson.Marshaler, bool) { // Check value receivers if v is not a pointer and pointer receivers if v is a pointer if v.Type().Implements(marshalerType) { return v.Interface().(encodingjson.Marshaler), true } // Check pointer receivers if v is not a pointer if v.Kind() != reflect.Ptr && v.CanAddr() { v = v.Addr() if v.Type().Implements(marshalerType) { return v.Interface().(encodingjson.Marshaler), true } } return nil, false } func toUnstructured(sv, dv reflect.Value) error { </ins> // Check if the object has a custom JSON marshaller/unmarshaller. <del> if st.Implements(marshalerType) { </del> <ins> if marshaler, ok := getMarshaler(sv); ok { </ins> if sv.Kind() == reflect.Ptr && sv.IsNil() { // We're done - we don't need to store anything. return nil } <del> marshaler := sv.Interface().(encodingjson.Marshaler) </del> data, err := marshaler.MarshalJSON() if err != nil { return err

Kubernetes version (use ): master. What happened: Unstructured converter from does not roundtrip . What you expected to happen: When I call on a which is backed by it should basically make a copy of the content (can be optimized btw using deep copy ) and not a json object with key. How to reproduce it (as minimally and precisely as possible): See the test in my branch Test output: Anything else we need to know: For some reason changed the argument types from to . I guess the intent is to make the converter a more generic tool. But based on the previous signature I assume I'm doing the right thing. Actually, my code works with and I don't know if the actual object behind the interface is an or not - I just pass it to the converter and it works for usual API types, but not for unstructured. Also it would be useful to have a more type-safe version of the converter (i.e. the previous signature) that just delegates to a lower-level converter that does not care about types.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/kind bug
/assign
Looked into this issue just now. It is caused by the pointer vs non-pointer receiver of the method - similar to . In detail: is passed as to when it calls passing as . if passed value implements (has method ) does not implement this interface, does. Because on it . So perhaps the fix is to do the dereferencing inside of the ? Or maybe the same way as - check if the value is addressable, get a pointer to the value and then see if it matches the interface (I dunno if it will work that way)? In any case, the check whether a value implements the interface should be done on both the value itself () and the pointer to the value (; if the value is addressable) to cover both valid cases. This is what stdlib package does. Right now it is broken - it dereferences the pointer and then checks if the thing implements the interface. Does it make sense?

Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
does this helps? Look like we need to do MarshalJSON on non-pointers for it to work. I didn't get into the details but if that is the case then making Unstructured's MarshalJSON() receiver, a object instead of pointer should fix your problem, right?
Only if we don't fix the actual cause of the bug which I described above. From the stack overflow question that is mentioned in the link you provided: This code is buggy - to work it should be passing into and into . Yes, json package tries to take the address of the passed (so a pointer receiver should satisfy the interface check) but it cannot take an address of a function argument ( will return false). So, pointer receiver will be invoked if either is a field in a struct (so that an address can be taken via reflection) and a struct is passed (either by value or via a pointer) to / OR is passed. Please take a look at - both issues have the same cause.
I'm happy to take a stab at it and submit a PR but I don't want to spend my time if it is not going to get merged - if we could merge first to make sure people who can approve/lgtm are happy with the approach taken. I strongly believe it is the right thing to do - that is how standard library does it and, well, it is just logical and makes sense :)


Question: }, InvolvedObject: api.ObjectReference{ Namespace: "bar", <ins> Kind:      "Pod", </ins> }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ <del> Name:      "test1", </del> <ins> Name:      "test2", </ins> Namespace: "aoeu-_-aoeu", }, InvolvedObject: api.ObjectReference{ Namespace: "aoeu-_-aoeu", <ins> Kind:      "Pod", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test3", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "v1", Kind:       "Node", }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test4", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "v1", Kind:       "Namespace", }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test5", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "extensions/v1beta1", Kind:       "NoKind", Namespace:  api.NamespaceDefault, }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test6", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "extensions/v1beta1", Kind:       "Job", Namespace:  "foo", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test7", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "extensions/v1beta1", Kind:       "Job", Namespace:  api.NamespaceDefault, }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test8", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "other/v1beta1", Kind:       "Job", Namespace:  "foo", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test9", Namespace: "foo", }, InvolvedObject: api.ObjectReference{ APIVersion: "other/v1beta1", Kind:       "Job", Namespace:  "foo", }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test10", Namespace: api.NamespaceDefault, }, InvolvedObject: api.ObjectReference{ APIVersion: "extensions", Kind:       "Job", Namespace:  "foo", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test11", Namespace: "foo", }, InvolvedObject: api.ObjectReference{ // must register in v1beta1 to be true APIVersion: "extensions/v1beta1", Kind:       "Job", Namespace:  "foo", }, }, true, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test12", Namespace: "foo", }, InvolvedObject: api.ObjectReference{ APIVersion: "other/v1beta1", Kind:       "FooBar", Namespace:  "bar", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test13", Namespace: "", }, InvolvedObject: api.ObjectReference{ APIVersion: "other/v1beta1", Kind:       "FooBar", Namespace:  "bar", }, }, false, }, { &api.Event{ ObjectMeta: api.ObjectMeta{ Name:      "test14", Namespace: "foo", }, InvolvedObject: api.ObjectReference{ APIVersion: "other/v1beta1", Kind:       "FooBar", Namespace:  "", </ins> }, }, false,

ValidateEvent() in checks for "Node" specifically, but should really check for "any non-namespaced object.
I'd like to fix this.
Harder than it sounds, because events may reference 3rd party objects for which we don't know this piece of information...