Question: responses: 200: body: <del> application/json: example: !include examples/service-list.json </del> <ins> example: !include examples/service-list.json </ins> post: description: Create a new service body: <del> json/application: schema: !include doc/service-schema.json example: !include examples/service.json </del> <ins> schema: !include doc/service-schema.json example: !include examples/service.json </ins> /{serviceId}: get:

Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Question: clone, err := conversion.NewCloner().DeepCopy(&volume) if err != nil { glog.Errorf("error cloning volume %q: %v", volume.Name, err) <ins> continue </ins> } volumeClone := clone.(*api.PersistentVolume) ctrl.storeVolumeUpdate(volumeClone)

As part of the Kubernetes upgrade workflow we have in place, we are upgrading kubernetes from 1.15.3 -1.18.15 (both versions and all versions in the middle are custom patched for different behavior of the and . None of them having anything to do with kube-proxy's IPVS), and as part of this upgrade we are also migrating the from to mode. We have calico 3.12 running on the cluster as part of the CNI stack. Once the pods are migrated from to mode by applying the new DaemonSet manifest, the cluster has a series of stale rules for chain left around that breaks the runtime state of the pods. have cluster running in HA mode (3 Node cluster) where all nodes of the cluster are running as leaders and take the workload as well. There is no non-leader Pods. have customized the kubernetes and calico infra to enable the usage of Link local IP address in range for internal usecases Most of the IPs and the names have been mangled. Stale IP table rules to be cleaned up once the pods and endpoints are stable with all endpoints in Ready address mode. down the pods on a given namespace to 0 (leave the service and Endpoints around) the control plane to 1.18.15 (take the intermediate hop path. 1.15.3 -1.16.7 -1.17.5 -1.18.15) apply the kube-proxy manifest provided above. up the pods scaled down in Step 1. Current workaround we have been putting it use the as an init container and run the and then bring up the in IPVS mode and that seem to take care of the stale entries (but this seem to be too much of a big hammer approach of cleaning up) Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools: kubeadmNetwork plugin and version (if this is a network-related bug):Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label area/kube-proxy Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig/network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
I don't think that the live migration between kube-proxy implementations iptables<-ipvs is supported, at least I never heard about that and seems tricky. you use to have all the historical context, what do you think?
We tried implementing a cleanup mode but it was problematic and removed. (see ) You can run but that's pretty coarse. Maybe we should add a value to that flag to say which mode to try to cleanup? PRs welcome. The simplest answer is to reboot.
Let's make this a docs issues - we should write this down
can we merge the above so that we can close this issue?
not just yet :-) wouldn't signal registration and calling clean-up would at least clean things up on graceful shutdown? Granted rules will not be there until kube-proxy is restarted.


Question: go_library( name = "k8s.io/apiserver/pkg/authentication/group", <del> srcs = ["k8s.io/apiserver/pkg/authentication/group/group_adder.go"], </del> <ins> srcs = [ "k8s.io/apiserver/pkg/authentication/group/authenticated_group_adder.go", "k8s.io/apiserver/pkg/authentication/group/group_adder.go", ], </ins> tags = ["automanaged"], deps = [ "//vendor:k8s.io/apiserver/pkg/authentication/authenticator",

When an auth proxy tries to proxy an anonymous request, the user.Info ends up with the group system:authenticated, which is wrong.


Question: package kuberuntime import ( <del> "fmt" </del> v1 "k8s.io/api/core/v1" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" "k8s.io/kubernetes/pkg/security/apparmor"

Hello, What happened: When we apply PodSecurityPolicy with rule: WIndows containers are still blocked instead we put: the pod is still blocked with logs: What you expected to happen: the Container should start How to reproduce it (as minimally and precisely as possible): Create Psp disableling Root Users: Create a pod Windows with runAsUserName: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux 7 (Core) / windows server 2019 version 1809 Kernel (e.g. ): 3.10.0-514.10.2.el7.x86_64 Others: /sig windows
/sig windows
/assign I will test it and then post the test results
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: set -e <ins> #clean all init/init.d/configs function do_backup_clean() { #backup all config files init_files=`ls init_conf` for i in $init_files do if [ -f /etc/init/$i ] then mv /etc/init/${i} /etc/init/${i}.bak fi done initd_files=`ls initd_scripts` for i in $initd_files do if [ -f /etc/init.d/$i ] then mv /etc/init.d/${i} /etc/init.d/${i}.bak fi done default_files=`ls default_scripts` for i in $default_files do if [ -e /etc/default/$i ] then mv /etc/default/${i} /etc/default/${i}.bak fi done # clean work dir if [ ! -d ./work ] then mkdir work fi cp -rf default_scripts init_conf initd_scripts work } </ins> function cpMaster(){ # copy /etc/init files <del> cp init_conf/etcd.conf /etc/init/ cp init_conf/kube-apiserver.conf /etc/init/ cp init_conf/kube-controller-manager.conf /etc/init/ cp init_conf/kube-scheduler.conf /etc/init/ </del> <ins> cp work/init_conf/etcd.conf /etc/init/ cp work/init_conf/kube-apiserver.conf /etc/init/ cp work/init_conf/kube-controller-manager.conf /etc/init/ cp work/init_conf/kube-scheduler.conf /etc/init/ </ins> # copy /etc/initd/ files <del> cp initd_scripts/etcd /etc/init.d/ cp initd_scripts/kube-apiserver /etc/init.d/ cp initd_scripts/kube-controller-manager /etc/init.d/ cp initd_scripts/kube-scheduler /etc/init.d/ </del> <ins> cp work/initd_scripts/etcd /etc/init.d/ cp work/initd_scripts/kube-apiserver /etc/init.d/ cp work/initd_scripts/kube-controller-manager /etc/init.d/ cp work/initd_scripts/kube-scheduler /etc/init.d/ </ins> # copy default configs <del> cp default_scripts/etcd /etc/default/ cp default_scripts/kube-apiserver /etc/default/ cp default_scripts/kube-scheduler /etc/default/ cp default_scripts/kube-controller-manager /etc/default/ </del> <ins> cp work/default_scripts/etcd /etc/default/ cp work/default_scripts/kube-apiserver /etc/default/ cp work/default_scripts/kube-scheduler /etc/default/ cp work/default_scripts/kube-controller-manager /etc/default/ </ins> } function cpMinion(){ # copy /etc/init files <del> cp init_conf/etcd.conf /etc/init/ cp init_conf/kubelet.conf /etc/init/ cp init_conf/flanneld.conf /etc/init/ cp init_conf/kube-proxy.conf /etc/init/ </del> <ins> cp work/init_conf/etcd.conf /etc/init/etcd.conf cp work/init_conf/kubelet.conf /etc/init/kubelet.conf cp work/init_conf/flanneld.conf /etc/init/flanneld.conf cp work/init_conf/kube-proxy.conf /etc/init/ </ins> # copy /etc/initd/ files <del> cp initd_scripts/etcd /etc/init.d/ cp initd_scripts/flanneld /etc/init.d/ cp initd_scripts/kubelet /etc/init.d/ cp initd_scripts/kube-proxy /etc/init.d/ </del> <ins> cp work/initd_scripts/etcd /etc/init.d/ cp work/initd_scripts/flanneld /etc/init.d/ cp work/initd_scripts/kubelet /etc/init.d/ cp work/initd_scripts/kube-proxy /etc/init.d/ </ins> # copy default configs <del> cp default_scripts/etcd /etc/default/ cp default_scripts/flanneld /etc/default/ cp default_scripts/kube-proxy /etc/default cp default_scripts/kubelet /etc/default/ </del> <ins> cp work/default_scripts/etcd /etc/default/ cp work/default_scripts/flanneld /etc/default/ cp work/default_scripts/kube-proxy /etc/default cp work/default_scripts/kubelet /etc/default/ </ins> } # check if input IP in machine list

I used the ubuntu-cluster setup scripts to build a mini testing environment. The first time I inputted a wrong config and when I reconfigured the environment by running the scripts again an error occured. I looked into the code and found the scripts hadn't deal with old configs. And of kube-proxy configs are not replaced.
Looks like this was fixed by .


Question: } CPUDetails := CPUDetails{} numCPUs := machineInfo.NumCores numPhysicalCores := 0 <ins> var coreID int var err error </ins> for _, socket := range machineInfo.Topology { numPhysicalCores += len(socket.Cores) for _, core := range socket.Cores { <ins> if coreID, err = getUniqueCoreID(core.Threads); err != nil { glog.Errorf("could not get unique coreID for socket: %d core %d threads: %v", socket.Id, core.Id, core.Threads) return nil, err } </ins> for _, cpu := range core.Threads { CPUDetails[cpu] = CPUInfo{ <del> CoreID:   core.Id, </del> <ins> CoreID:   coreID, </ins> SocketID: socket.Id, } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CPU Manager uses topology from cAdvisor which uses which is different from in terms of coreID's. assigns coreID's in unique manner (platform-wise) whereas has coreID's from CPU registers which are unique only on socket level. This causes some unexpected cpu assignment on platforms with more than one socket. All tests written all "false positive" Example output: From What you expected to happen: Fix either by changing the way CPU manger sees topology (use ) or redesign CPU Manager to handle coreID's as socket only unique How to reproduce it (as minimally and precisely as possible): kubelet on dual socket node G pod that requests 2 cores pod gets 2 cores on different sockets
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: err = framework.WaitTimeoutForPodRunningInNamespace(config.client, pod.Name, pod.Namespace, framework.PodStartShortTimeout) Expect(err).To(HaveOccurred()) <del> checkPodEvents(config, pod.Name, ep) </del> cleanupLocalVolumes(config, []*localTestVolume{testVol}) })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: generatorName := cmdutil.GetFlagString(cmd, "generator") if len(generatorName) == 0 { <ins> client, err := f.Client() if err != nil { return err } resourcesList, err := client.Discovery().ServerResources() if err != nil { // this cover the cases where old servers do not expose discovery resourcesList = nil } </ins> if restartPolicy == api.RestartPolicyAlways { <del> generatorName = "deployment/v1beta1" </del> <ins> if contains(resourcesList, v1beta1.SchemeGroupVersion.WithResource("deployments")) { generatorName = "deployment/v1beta1" } else { generatorName = "run/v1" } </ins> } else { <del> generatorName = "job/v1" </del> <ins> if contains(resourcesList, batchv1.SchemeGroupVersion.WithResource("jobs")) { generatorName = "job/v1" } else if contains(resourcesList, v1beta1.SchemeGroupVersion.WithResource("jobs")) { generatorName = "job/v1beta1" } else { generatorName = "run-pod/v1" } </ins> } } generators := f.Generators("run")

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: glog.V(4).Infof("Disk %s is already attached to instance %s", volumeID, instanceID) return volume.ID, nil } <del> glog.V(2).Infof("Disk %s is attached to a different instance (%s), detaching", volumeID, volume.AttachedServerId) err = os.DetachDisk(volume.AttachedServerId, volumeID) if err != nil { return "", err } </del> <ins> errmsg := fmt.Sprintf("Disk %s is attached to a different instance (%s)", volumeID, volume.AttachedServerId) glog.V(2).Infof(errmsg) return "", errors.New(errmsg) </ins> } startTime := time.Now()

In Cinder cloud provider openstack Inside of AttachDisk function, it issues DetachDisk if it find out the the volume is currently attached to a different instance. This is very dangerous because it is very likely the volume is still mounted to that instance. The correct behavior should be fail this AttachDisk call. K8s attachdetach controller should handle unmount and detach volume safely.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
cc


Question: nodeRef:      nodeRef, } <del> err := im.start() if err != nil { return nil, fmt.Errorf("failed to start image manager: %v", err) } </del> return im, nil } <del> func (im *realImageManager) start() error { </del> <ins> func (im *realImageManager) Start() error { </ins> // Initial detection make detected time "unknown" in the past. var zero time.Time err := im.detectImages(zero)

I'm trying to start 100-node cluster. I got the following error in kubelet: E0505 08:31:55. 5066 ] Failed to create kubelet: failed to initialize image manager: failed to start image manager: dial unix connection refused After ssh-ing I checked that docker is working by running . I guess it must be some kind of a race where kubelet starts before docker is ready. When I start 3-node cluster it works just fine. This happened to me 3 times in a row. Please add this to 1.0 milestone if necessary.
I've just exactly the same problem (when creating a 100-node cluster)
This is a bug in our initialization logical. We initialize image manager which needs to talk to docker at very early stage before docker is up yet. Later kubelet does a wait to make sure docker is up. We should either delay the initialization of image manager or handle docker unavailable better. Kubelet will be restarted by monit eventually, but this definitely slow down node initialization. I mark this for 1.0. Should be a simple fix if we just simply delay the initialization of image manager.
I'll go ahead and fix it (since I am mainly to blame :) )
Changed to P0 because asked us to only add P0 issues to v1.0 milestone.


Question: func findDiskByLun(lun int, io ioHandler, exec mount.Exec) (string, error) { return "", nil } <del> func formatIfNotFormatted(disk string, fstype string, exec mount.Exec) { } </del>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When use an existing azure disk(also called ) in pod, if that disk has partitions, the disk will be formatted in the pod mounting. What you expected to happen: How to reproduce it (as minimally and precisely as possible): create a disk with source from an os disk, e.g. that disk is already formatted before pod could access it Anything else we need to know?: Root cause: current diskLooksUnformatted func is using command to check whether disk is formatted or not while common k8s code() is using to detect disk format, could detect multiple partitions so if we use common k8s code(), following error will be returned for mulitple partition disks, which is expected: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
cc
iirc we don't support partitions on Azure volumes. There was a specific hang long ago. in all cases i think should be safe to based check now via the underlaying api
thanks for the info, I am writing this PR, and also testing carefully. And this issue depends on , could you review it first? Thanks.


Question: "k8s.io/client-go/openapi" cachedopenapi "k8s.io/client-go/openapi/cached" restclient "k8s.io/client-go/rest" <ins> "k8s.io/klog/v2" </ins> ) type cacheEntry struct {

I think this got stale and closed because it was posted on but is still very relevant: The created by memory.NewMemCacheClient(...) returns an error when calling ServerResourcesForGroupVersion if the APIResources list returned is empty, as seen here: Our particular case came up due to the API group being registered with the API server, but having no resources. If using like to discover with zero current resources there should not be an error returned. Do a discovery with a against a Kubernetes cluster that has at least one with zero resources. The condition on the server that triggers the described issue can be expressed as e.g.: I would be good to get to know the reasoning behind the emptiness check in the , maybe there is good reason for it. <detailsClient Version: v1.20.15 Server Version: v1.22.8-gke.202 </details<detailsGoogle Cloud (GKE) </detailsNo response No response No response No response
/sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
Reviving this issue based on This behavior has become quite annoying for the custom/external metrics aggregated APIs since after , client-go is now producing a very spammy error log when an aggregated API is found without resource. This is a very common scenario with the metric APIs since metrics are dynamically registered as Resources to the aggregated APIs so on startup the aggregated apiserver may not expose any resource and that's completely normal. It is even more common for implementations of these APIs that register metric Resources from CRDs as CRs tend to be created later than APIServices and the aggregated apiserver. I can see a few path forward to improve the situation for us: the error if we consider that having no resources exposed by an aggregated API is acceptable the verbosity of the log to just show it in debugging scenarios somehow update to defer registration until we have at least one Metric resource. could you perhaps share your insight on that one please? /remove-lifecycle stale
Hello folks! Any update on this topic? The message is really spammy
Removing triage accepted manually since there was no update on this for more than 6 months. /remove-triage accepted
For context, I traced it back and check has been considered an error since the first iteration of the in-memory discovery client in : It is unexpected for a built-in or CRD groupversion to have no resources, but for aggregated-apiservice I can see it being possible that the object is created and set up for discovery before the aggregated-apiservice is ready.
I think this is which introduces (but maybe I'm wrong). There is also in that repo about the same issue after upgrading the kube-client pkg. in custom-metrics repo be correct from Kubernetes pov? IMO we need to find a solution soon or the amount of noise will increase once people bump their deps and tooling (, kubectl is affected, and any tool which uses kube-client is affected)
/triage accepted


Question: # First build a release $(dirname $0)/../release/release.sh <ins> if [ "$?" != "0" ]; then echo "Building a release failed!" exit 1 fi </ins> # Now push this out to the cluster $(dirname $0)/../cluster/kube-push.sh

dev-build-and-{push|up}.sh scripts silently push an old version.


Question:
 If the webhook is running within the cluster, then you should use service.

  If there is only one port open for the service, that port will be used. If there are multiple ports open, port 443 will be used if it is open, otherwise it is an error.
	   Port 443 will be used if it is open, otherwise it is an error.

true

v1beta1.ServiceReference

What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object


Question: } createNode2NormalPodEviction := func(client clientset.Interface) func() error { return func() error { <ins> zero := int64(0) </ins> return client.Policy().Evictions("ns").Evict(&policy.Eviction{ TypeMeta: metav1.TypeMeta{ APIVersion: "policy/v1beta1",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set.

Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


Question: // Until is a wrapper around workqueue.ParallelizeUntil to use in scheduling algorithms. // A given operation will be a label that is recorded in the goroutine metric. func (p Parallelizer) Until(ctx context.Context, pieces int, doWorkPiece workqueue.DoWorkPieceFunc, operation string) { <ins> goroutinesMetric := metrics.Goroutines.WithLabelValues(operation) </ins> withMetrics := func(piece int) { <del> metrics.Goroutines.WithLabelValues(operation).Inc() defer metrics.Goroutines.WithLabelValues(operation).Dec() </del> <ins> goroutinesMetric.Inc() </ins> doWorkPiece(piece) <ins> goroutinesMetric.Dec() </ins> } workqueue.ParallelizeUntil(ctx, p.parallelism, pieces, withMetrics, workqueue.WithChunkSize(chunkSizeFor(pieces, p.parallelism)))

We observed degradation in scheduling throughput, putting it under 100 pods/s pprof points to ! /sig scheduling /kind regression 100 pods/s N/A No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/assign cc
/triage accepted
Oh.. sorry for the degradation and thanks for working on this. This time, as goroutine metric is called every time goroutine is created, we observed the clear performance degradation. But, there are actually many other metrics which is stored directly by calling everytime. So, for the future direction, is it better (slightly?) for the performance to move all other metrics to store via ? (like metric)
No need to be sorry. Just tagging you as FYI so that we are aware of this kind of performance penalties in future reviews :)
No, this code is too complex to be worth it. The rest of the metrics usage doesn't show up in pprofs, so no need to optimize further for now. We just need to be diligent not to call within critical routines when they all share the same values.


Question: } <ins> func TestDevicesToAllocateConflictWithUpdateAllocatedDevices(t *testing.T) { podToAllocate := "podToAllocate" containerToAllocate := "containerToAllocate" podToRemove := "podToRemove" containerToRemove := "containerToRemove" deviceID := "deviceID" resourceName := "domain1.com/resource" socket := filepath.Join(os.TempDir(), esocketName()) devs := []*pluginapi.Device{ {ID: deviceID, Health: pluginapi.Healthy}, } p, e := esetup(t, devs, socket, resourceName, func(n string, d []pluginapi.Device) {}) waitUpdateAllocatedDevicesChan := make(chan struct{}) waitSetGetPreferredAllocChan := make(chan struct{}) p.SetGetPreferredAllocFunc(func(r *pluginapi.PreferredAllocationRequest, devs map[string]pluginapi.Device) (*pluginapi.PreferredAllocationResponse, error) { waitSetGetPreferredAllocChan <- struct{}{} <-waitUpdateAllocatedDevicesChan return &pluginapi.PreferredAllocationResponse{ ContainerResponses: []*pluginapi.ContainerPreferredAllocationResponse{ { DeviceIDs: []string{deviceID}, }, }, }, nil }) testManager := &ManagerImpl{ endpoints:             make(map[string]endpointInfo), healthyDevices:        make(map[string]sets.Set[string]), unhealthyDevices:      make(map[string]sets.Set[string]), allocatedDevices:      make(map[string]sets.Set[string]), podDevices:            newPodDevices(), activePods:            func() []*v1.Pod { return []*v1.Pod{} }, sourcesReady:          &sourcesReadyStub{}, topologyAffinityStore: topologymanager.NewFakeManager(), } testManager.endpoints[resourceName] = endpointInfo{ e: e, opts: &pluginapi.DevicePluginOptions{ GetPreferredAllocationAvailable: true, }, } testManager.healthyDevices[resourceName] = sets.New[string](deviceID) testManager.podDevices.insert(podToRemove, containerToRemove, resourceName, nil, nil) go func() { <-waitSetGetPreferredAllocChan testManager.UpdateAllocatedDevices() waitUpdateAllocatedDevicesChan <- struct{}{} }() set, err := testManager.devicesToAllocate(podToAllocate, containerToAllocate, resourceName, 1, sets.New[string]()) assert.NoError(t, err) assert.Equal(t, set, sets.New[string](deviceID)) } </ins> func TestGetDeviceRunContainerOptions(t *testing.T) { res1 := TestResource{ resourceName:     "domain1.com/resource1",

It is suspected to be a concurrent bug of kubelet. When we delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time, kubelet may panic on as following. I think it's a concurrency issue. a pod with GPU request, then kubelet will allocate resource for it. Kubelet try call device plugin to choose preferred device for it. It will unlock the mutex!. we delete the last pod that requires GPU. the second pod with GPU request, kubelet will allocate resource for it also. Kubelet will call to recalculate device (delete allocated device if pod deleted). It will update the allocatedDevices map of ManagerImpl, because of mutex is unlock by step 1. Kubelet continues with the first pod. If allocatedDevices has changed and GPU device is nil, kubelet will panic in the closure allocateRemainingFrom. Don't panic. Delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time. No response <details</details<detailsexternal </details<details</detailsNOOP <detailsContainerd </detailsNOOP
/assign
/sig node
Is this reproduced in the master branch?
It is possible from the code. I will try to reproduce it later. Hope someone can give me some ideas also.
/cc Could you take a look? The issue scenario looks real to me.
hi, this looks legit, but OTOH a key assumption in the resource management allocation is that all the pods are admitted serially (e.g. NOT concurrently) one after the another. In other words, a key assumption is we never try allocating resources while another allocation is ongoing, e.g. not completed. So is not obvious to me which flows are conflicting. [EDIT] could still be if there is a client accessing the podresources API:
/triage accepted /priority important-soon I think the issue described here is real but I'm not sure how often it appears. Panics are always bad, but how often do we know this is happening? Let's start with as priority [EDIT] subejct to further adjustments as the investigation progresses.
Sure. Kubelet will allocate resources to pods when admitting. Admit is sequential.

As long as it is not device re-allocate, the problem will not occur. Podresources API is suspicious.
I deployed the dcgm exporter and it does call the interface of podresources.
This is a genuine and reproducible issue. We have observed the kubelet crashing due to such a panic when inserting a device into a map (): In our observation, right after deleting the pod, the podresources API was called. This removed the state of for the GPU resource. This happened when the was called as soon as the mutex got unlocked in the . Moreover, such a race seems being triggered when Device Plugin periodically calls podresources API.
Thanks, this information matches with other findings. The podresources endpoint intentionally triggers to report the current up to date allocation; otherwise the device manager would clean up lazily the allocation state when a new admission is attempted. We will work on a fix.
Just to add we've also seen this a few times on 1.23 and 1.24 clusters in EKS with the nvidia-device-plugin + DCGM-Exporter
This looks like a real problem. I unit tests to verify that the PR works.
not taking for 1.29, but may consider for a cherry-pick later. Depending on the fix complexity
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority important-longterm because low capacity
/remove-priority important-soon


Question: . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" apps "k8s.io/api/apps/v1beta1" <ins> appsv1beta2 "k8s.io/api/apps/v1beta2" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" klabels "k8s.io/apimachinery/pkg/labels"

What happened: There are no e2e and integration tests for statefulset scale subresource. What you expected to happen: (1) e2e test should exist BEFORE code freeze (2) integration test should exist AFTER 1.8 release branches cut Anything else we need to know?: Refer PR for more info. We should NOT remove existing e2e test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: } } // Delete dummy interface created by ipvs Proxier. <del> nl := NewNetLinkHandle() </del> <ins> nl := NewNetLinkHandle(false) </ins> err := nl.DeleteDummyDevice(DefaultDummyDevice) if err != nil { klog.Errorf("Error deleting dummy device %s created by IPVS proxier: %v", DefaultDummyDevice, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area ipv6 /area ipvs /sig network What happened: In an ipv6-only cluster with proxy-mode=ipvs the entries for NodePort in ipvs are created for ipv4 addresses. The target's (if there are any) consists of ipv6 addresses (wrongly) interpreted as ipv4; Attempts to use the NodePort with ipv6 addresses are black-holed. What you expected to happen: Ipv6 addresses to the nodes should be used for NotePort and the LB-targets shall be the ipv6 address of the pods. Traffic to NodePorts shall be distributed among the (ipv6) endpoints. How to reproduce it (as minimally and precisely as possible): Setup an ipv6-only cluster Create a service with type with endpoints Try to connect to the NodePort Do Anything else we need to know?: The nodes have both ipv4 and ipv6 addresses and it is possible that if the nodes only had ipv6 addresses it would work (I havn't tested). But even in an ipv6-only k8s cluster the nodes almost always need ipv4 addresses for other purposes. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm VMs on Ubuntu 18.04.01 LTS OS (e.g. from /etc/os-release): Own BusyBox based system Kernel (e.g. ):Install tools: None Others:
The NodeIPs() function seem to return only ipv4 addresses.
I can't see why (yet) since is used in the GetLocalAddresses function
/cc Please give a hint before I dive into the netlink code.
This issue is used as an example here;
The functions filters on Src != nil . But ipv6 addresses seems to have Src == nil always. As it is this function will never return any ipv6 addresses.


Question: cleanupSteps = append(cleanupSteps, func() { framework.ExpectNoError(sr.CleanupResource(f.Timeouts)) }) <del> vs := sr.Vs vsc := sr.Vsclass </del> err = e2epv.WaitForPersistentVolumeClaimPhase(v1.ClaimBound, cs, pvcNamespace, pvcName, framework.Poll, f.Timeouts.ClaimProvision) framework.ExpectNoError(err)

The changes to support snapshot tests for Generic Ephemeral Volumes generated some duplicated code in the file as commented here: , it'd be great to create functions that are reused in both and tests. Things that are needed: A kubernetes dev cluster (I follow to create a GCE cluster) Knowledge about how to /sig storage /area testing /good-first-issue Improve readability of the storage VolumeSnapshot tests
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi Could you please give me some more details about what needs to be done in this issue?
I modified the description, it's all about making changes in a single file creating functions that reduce the duplicated code following the thread also linked in the description.
Thanks!

Hi, I'm currently in the process of working on this issue. I'm adding the functions to the file.
is this issue solved because now I am unable to see the duplication which I saw day before yesterday.
Hi When I try to run the snapshottable tests using and I get 8 of the following failures: I have setup and as their documentation suggested, but seem to not be able to run the tests. Do you have an idea why this happens? Thanks!
what was the command that you used? The csi-hostpath driver is embedded into the kubernetes codebase so you don't need to add anything else to your kind? cluster.
The command I used to run the snapshottable tests in a local kind cluster is: . Does that seem OK?
do you have an idea about why this happens? Thanks!
hey sorry for the late response and thanks for the reminder, I always run my tests with and with a GCE cluster (not a kind cluster) for example following . The VolumeSnapshot feature requires a CSI Driver, in our e2e tests we use a CSI Driver that's embedded in the kubernetes codebase called csi-hostpath, from the logs I saw that it used csi-hostpath which I think it's ok, ideally we should test it with a real CSI Driver like PD CSI Driver but that's going to provision storage in GCP. Have you tried with ? I think that there's more info in that page about kubetest than kubetest2
Yes, I tried , but got the same errors as well. Do you think it's maybe a problem with ?
/assign
/assign
hey and you guys have solved this issues or still working on it, because i'm considering to solve this issue if you haven't solved it
/assign
thank you for signing up for this task, I believe should work, we're waiting for additional review


Question: }) AfterEach(func() { <ins> By("Stop and wait for recycle goroutine to finish") close(stopCh) wg.Wait() By("Clean all PVs") </ins> for nodeName, localVolumes := range allLocalVolumes { By(fmt.Sprintf("Cleaning up %d local volumes on node %q", len(localVolumes), nodeName)) cleanupLocalVolumes(config, localVolumes) } cleanupStorageClass(config) <del> By("Wait for recycle goroutine to finish") close(stopCh) wg.Wait() </del> }) It("should be able to process many pods and reuse local volumes", func() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-serial Which test(s) are failing: [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes Since when has it been failing: since introduction Testgrid link: https://k8s- Reason for failure: Anything else we need to know:
/assign
/reopen 1 failure out of recent 4 builds since is merged, will make it stable
Reopened this issue. <detailsIn response to : /reopen 1 failure out of 4 builds since is merged, will make it stable Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This failure is because kubelet is restarted serval times during testing. fail time of test: Note that after , the rest 3 pods cannot finish. This is because kubelet is restarted serval times in this period of time. Output of : The rest 3 jobs was assigned to this node, it cannot finished because kubelet is restarted repeatedly. Because all tests in this job is run serially. I'm not sure what caused kubelet to restart during this test is running. kubelet.service is configured with following configurations: With , kubelet may be restarted by clean/unclean signal, exit code or watchdog, timeout. And may kill it when it failed to , here is kubelet-
I guess these kubelet restarts are not expected, but fixing it is beyond my ability. One workaround is to increase timeout to tolerate them, what do you think? Because we check pod conditions periodically, it will not slow down test.
Maybe we need to get some help from node team to figure out why kubelet is restarting. cc
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
This test has not flaked in almost 2 weeks. /close
Closing this issue. <detailsIn response to : This test has not flaked in almost 2 weeks. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fi } <del> # Verify and find the various tar files that we are going to use on the server. # # Vars set: #   SERVER_BINARY_TAR #   SALT_TAR function find-release-tars { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" >&2 exit 1 fi SALT_TAR="${KUBE_ROOT}/server/kubernetes-salt.tar.gz" if [[ ! -f "$SALT_TAR" ]]; then SALT_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-salt.tar.gz" fi if [[ ! -f "$SALT_TAR" ]]; then echo "!!! Cannot find kubernetes-salt.tar.gz" >&2 exit 1 fi } </del> # Use the gcloud defaults to find the project.  If it is already set in the # environment then go with that. #

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: klog.V(4).Infof("Updating %s", castObj.Name) c.enqueue(castObj) } <ins> func (c *ConditionController) deleteCustomResourceDefinition(obj interface{}) { castObj, ok := obj.(*apiextensions.CustomResourceDefinition) if !ok { tombstone, ok := obj.(cache.DeletedFinalStateUnknown) if !ok { klog.Errorf("Couldn't get object from tombstone %#v", obj) return } castObj, ok = tombstone.Obj.(*apiextensions.CustomResourceDefinition) if !ok { klog.Errorf("Tombstone contained object that is not expected %#v", obj) return } } c.lastSeenGenerationLock.Lock() defer c.lastSeenGenerationLock.Unlock() delete(c.lastSeenGeneration, castObj.Name) } </ins>

/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


Question: docs/user-guide/kubectl/kubectl_set_image.md docs/user-guide/kubectl/kubectl_set_resources.md docs/user-guide/kubectl/kubectl_set_selector.md <ins> docs/user-guide/kubectl/kubectl_set_serviceaccount.md </ins> docs/user-guide/kubectl/kubectl_set_subject.md docs/user-guide/kubectl/kubectl_taint.md docs/user-guide/kubectl/kubectl_top.md

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: package aws import ( <ins> "fmt" </ins> "testing" "github.com/aws/aws-sdk-go/aws" <ins> "github.com/aws/aws-sdk-go/service/ec2" </ins> ) func TestElbProtocolsAreEqual(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After a rolling update of the cluster, NLB kept failing health checks because security group did not have rules for new EC2 instance. What you expected to happen: Security group should have been updated with both TCP rules for NLBs to work correctly. How to reproduce it (as minimally and precisely as possible): Create a service object with type LoadBalancer and adding the annotation for NLBs. FWIW, I had 6 services and each had their own NLBs. It’s possible that the bug happens when there are multiple NLBs causing multiple AWS updates. Perform a rolling update of the instance group. This causes the instances to be terminated and new ones to be brought up. AFAICT, this should result in two changes to the security group per EC2 instance. When the instance is terminated, the rules for that nodePort should be deleted. When the new node is created, the rule should be again. However, after the rolling update, the rule doesn’t get . As a result, the NLB’s health checks fail. Anything else we need to know?: As mentioned earlier, this happened on a cluster where there were multiple NLBs. This doesn’t seem to be a deterministic failure. The log has the following error message:I manually edited the Service object to see if it would simply retrigger an update on the cloud resources. It actually did. The rules got to the security group. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kops 1.8.1 Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you find any workarounds? we've run into this issue, where the security group rules got deleted. Specifically, we found that the security group missed the following: 1) TCP rule to allow traffic from NLB to Node Port 2) TCP rule to allow traffic from NLB to Healthcheck Port We these manually back in to fix, but were surprised these were deleted in the first place.

This stackoverflow also is relevant:
One workaround was to simply refresh the Kubernetes service object. We /updated a label in the Kubernetes service object and found that it would make the corresponding changes in AWS.
thanks have you been able to root cause why the rule doesn't get ? We're trying to re-produce the failure ourselves, too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Any update on this bug fix release please?
Sorry to bother you with commenting on this issue. But I cannot find out in what version of Kubernetes this is expected to be released / was released. Any clarification highly appreciated.


Question: accessModes: - ReadWriteOnce hostPath: <del> path: /tmp/pv-2 </del> <ins> path: /tmp/data/pv-2 </ins>

Version: Running this example on RHEL 7.2 (assume same for centos), and using hostPath, I get a permissions error when the mysql container does a . If I set selinux to permissive on the host it works. If I manually run the container with it works. I was not able to get it to work when adding to the deployment. My fix was to locate the PVs in by changing the local- file, and doing . I this comment to local- If this issue is valid I will create a quick PR to doc this selinux solution.
See pr


Question: } serviceCIDR, err = c.client.NetworkingV1alpha1().ServiceCIDRs().Create(context.Background(), serviceCIDR, metav1.CreateOptions{}) if err != nil && !apierrors.IsAlreadyExists(err) { <del> klog.Infof("error creating default ServiceCIDR: %v", err) </del> c.eventRecorder.Eventf(serviceCIDR, v1.EventTypeWarning, "KubernetesDefaultServiceCIDRError", "The default ServiceCIDR can not be created") <del> return </del> <ins> return err </ins> } <del> c.setReady() </del> c.syncStatus(serviceCIDR) <del> } func (c *Controller) setReady() { select { case <-c.readyCh: default: close(c.readyCh) } </del> <ins> return nil </ins> } func (c *Controller) syncStatus(serviceCIDR *networkingapiv1alpha1.ServiceCIDR) {

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


Question: "github.com/onsi/gomega" v1 "k8s.io/api/core/v1" <ins> discoveryv1 "k8s.io/api/discovery/v1" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields"

Kubernetes e2e suite: [It] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should reject everything except leases The job was run on Aug 5th. I had a PR fixing typos in comments of kubelet source code. It is not related to api-machinery ipv6 but the test failed. The re-test passed. No response No response It seems the test is recently . /sig api-machinery /sig testing
It is failing considerably , it seems it fails to connect to the webhook that matches the time the test was I don't fully understand the test but it seems that is interacting with the cluster, I don't have clear if the deny policy affects everything and this is causing the race with the pod that implements the webhook /priority critical-urgent /release blocker /cc
/assign
cc
Debugged with Joe offline, paste for reference click open stderr on the failure to get the timeline
still failing after merge of
/triage accepted
It's suspicious that of all the admission webhook tests that start up a webhook pod, this is the one that flakes. I'll look more closely.
Looks like the webhook in this test may be configured to deny requests except leases globally in the cluster once enabled., which might create a race where, if the webhook is registered first , the controller setting up the webhook pod is denied requests by the webhook. I'll scope the webhook matching to the test namespace per offline suggestion.
Please let me know if I can be of help debugging this.Have free cycle and would like to contribute.
/assign
I can still see references to the expression that gives error Example run: Could you help confirm that the tests are completely removed? Thank you. cc
I think this is another test that is impacting the rest of the tests. Probably because the tests are running in parrallel. We might want to scope things down. Currently working on a fix.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
this is a new bug, please review all the test that were and verify they are scoped to the custom created namespaces by the test framework, and if not, please tag them as cc:
/cc
I assume this could be closed with merged? /cc
Looks like the exact flak as reported in this issue has been resolved. /close
Closing this issue. <detailsIn response to : Looks like the exact flak as reported in this issue has been resolved. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: */ // +k8s:conversion-gen=k8s.io/kubernetes/pkg/apis/batch <ins> // +k8s:conversion-gen=k8s.io/kubernetes/pkg/apis/batch/v1 </ins> // +k8s:conversion-gen-external-types=k8s.io/api/batch/v1beta1 // +k8s:defaulter-gen=TypeMeta // +k8s:defaulter-gen-input=../../../../vendor/k8s.io/api/batch/v1beta1

it looks like something is up on master. Clean checkout, touch , then -- produces a delta.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
That comes out clean. ` That comes out clean. That dir, and who knows how many others, is missing some metadata to allow conversions
looks like it might be the v2alpha1 conversion referencing a v1 type... if the generation processes both the v1 and v2alpha1 packages, it finds the v1 conversion. if it only processes the v2alpha1 package, it doesn't and reverts to the reflection-driven path
this breaks: this works: given a clean build leaves no diff, I think master is fine. the script looks like it doesn't include all required dependencies to rebuild a single package
The tool doesn't know - each pkg i supposed to define enough to make itself hermetic
That are not many input dirs. The generator does not see any custom conversion funcs in v1. As expected, adding v1 to the inputs gives the correct output:
/assign fixed in


Question: // Config is a struct containing all arguments for creating a pod. // SELinux testing requires to pass HostIPC and HostPID as boolean arguments. type Config struct { <del> NS                  string PVCs                []*v1.PersistentVolumeClaim PVCsReadOnly        bool InlineVolumeSources []*v1.VolumeSource IsPrivileged        bool Command             string HostIPC             bool HostPID             bool SeLinuxLabel        *v1.SELinuxOptions FsGroup             *int64 NodeSelection       NodeSelection ImageID             int </del> <ins> NS                     string PVCs                   []*v1.PersistentVolumeClaim PVCsReadOnly           bool InlineVolumeSources    []*v1.VolumeSource IsPrivileged           bool Command                string HostIPC                bool HostPID                bool SeLinuxLabel           *v1.SELinuxOptions FsGroup                *int64 NodeSelection          NodeSelection ImageID                int PodFSGroupChangePolicy *v1.PodFSGroupChangePolicy </ins> } // CreateUnschedulablePod with given claims based on node selector

We need to add e2e that verifies volume permissions in pod with different values of fsGroupChangePolicy. /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: clock.Step(time.Minute * 1) ok, _ = pleg.Healthy() assert.True(t, ok, "pleg should be healthy") <ins> // Advance by relistThreshold without any relisting. pleg should be unhealthy // because it has been longer than relistThreshold since a relist occurred. clock.Step(relistThreshold) ok, _ = pleg.Healthy() assert.False(t, ok, "pleg should be unhealthy") </ins> } func TestRelistWithReinspection(t *testing.T) {

When the kubelet first starts the folllowing error is emitted to the logs This is because the relist has not yet run which results in a duration that is just the time since the zero time instant () xref /sig node
/assign
/approve /lgtm
help if i approve the pr and not the issue ;-)


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "path" "regexp"

[ ] Promotion created According to this APIsnoop query, there are still some remaining ServiceAccount endpoints which are untested. Create a ServiceAccount with a static label Create a Secret Patch the ServiceAccount with a new Label and a new Secret Get the ServiceAccount to ensure it's patched List all ServiceAccounts in all Namespaces find the ServiceAccount(1) ensure that the ServiceAccount is found and is patched Delete Namespaced ServiceAccount(1) via a Collection with a LabelSelector Delete the Secret Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Fine with moving forward You don't need to check the status of the secret as part of the test. In other places we check that the resource in question happens, we don't have to follow.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Write CoreV1ServiceAccount resource lifecycle test+promote - +3
/close close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close close this Issue in favour of and place in In Progress Issues on org dashboard </details


Question: go controller.Run(1, stopCh) } <ins> for _, sniCert := range s.SNICerts { if notifier, ok := sniCert.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } if controller, ok := sniCert.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(); err != nil { return nil, err } go controller.Run(1, stopCh) } } </ins> // runonce to be sure that we have a value. if err := dynamicCertificateController.RunOnce(); err != nil {

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return } <del> if immediate { jm.queue.Forget(key) </del> <ins> backoff := time.Duration(0) if !immediate { backoff = getBackoff(jm.queue, key) </ins> } <del> backoff := getBackoff(jm.queue, key) </del> // TODO: Handle overlapping controllers better. Either disallow them at admission time or // deterministically avoid syncing controllers that fight over pods. Currently, we only

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: if feature.DefaultFeatureGate.Enabled(features.JobReadyPods) { ready = pointer.Int32(countReadyPods(activePods)) } <del> // Job first start. Set StartTime and start the ActiveDeadlineSeconds timer // only if the job is not in the suspended state. </del> <ins> // Job first start. Set StartTime only if the job is not in the suspended state. </ins> if job.Status.StartTime == nil && !jobSuspended(&job) { now := metav1.Now() job.Status.StartTime = &now <del> // enqueue a sync to check if job past ActiveDeadlineSeconds if job.Spec.ActiveDeadlineSeconds != nil { klog.V(4).Infof("Job %s has ActiveDeadlineSeconds will sync after %d seconds", key, *job.Spec.ActiveDeadlineSeconds) jm.queue.AddAfter(key, time.Duration(*job.Spec.ActiveDeadlineSeconds)*time.Second) } </del> } var manageJobErr error

Prior to 1.24.0 setting in a Job object () would kill the Job and Pods if they ran longer than the integer specified. I tested today with 1.24.0 and it's not working. The Job defined below (for testing) should be killed 10 seconds, but isn't. It's actually being terminated after about 55 seconds. The Job to be killed after 10 seconds as per . It's worked like this in 1.23.x. On a cluster running 1.24.0 with the following YAML file called . Create the Job. On Kubernetes clusters earlier than 1.24.0 it would fail after 10 seconds and the following command should show in the events 10 seconds after the Pod is created. In K8s 1.24.0 it's terminated after about a minute instead of 10 seconds. It works as expected in 1.23.6 and other point versions of 1.23. It works as expected, and documented, in all versions of 1.23 I've tested on. Docs: <details</details<detailsAll and also local clusters. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig apps
I just searched for at locally and could not see any logic that compares running time with and marks a pod for termination. Do I need to look somewhere else?
It's here Here's the quote: "Another way to terminate a Job is by setting an active deadline. Do this by setting the field of the Job to a number of seconds. The applies to the duration of the job, no matter how many Pods are created. Once a Job reaches , all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded." I may be misunderstanding it, but I don't think I am as it works the way described in older versions. The Job owns the Pod, you set on the Job to a value, let's say . This will cause the Job to be killed if it's still running after 10 seconds. As the Job owns the Pod(s) it cleans them up as well. This is exactly how it works in previous versions.

The job controller for sure has the code for cleanup on active deadline still in place +1 from me as well. The behavior for sure is not the same on 1.23.x vs 1.24.0 using the same job manifest. And the documentation v/s the behavior is not the same.
Seem to be a side effect of feature gate. Disabling that gives me the old behavior as in 1.23.x You can get the same behavior even in 1.23.x with the feature flag enabled.
The weird bit as you see is the pod actually completes but when it processes the completion event it notices the activeDeadline value and marks deadline Exceeded while the pod actually ran to completion
/assign
I think I have figured out why this happened. It is a side effect of feature gate (thanks ). : when we call with multiple times, the workqueue will only keep the duration that will be triggered nearest from now. (when there is no feature gate or do not enable it): job-controller mainly relies on the first time sync to add the job after its : (unless the field has been changed then it will be add after ). : the feature gate makes to be and it will be add after for the job when a pod of it has been updated. Then the duration which has been after in the first time sync will be overwrite to . And after the next syncJob, there will be no one to addAfter the job again. WDYT
I have came to the same conclusion as you are based on what is in the and . So, I did a dirty prototype of tracking the activeDeadline in an entirely different to avoid conflicts and it seem to be working well. If this is an acceptable change to and I would be more than happy to raise a Draft PR for review along with some additional UT to cover the active deadline cases
Emm... Usually it is not suggested to have multiple active queues in one controller, which will cause an object to be handled by multiple workers at the same time and also make the progress more complicated. My suggestion is do at the end of syncJob every time an unfinished job is reconciled, the duration should be . This is clear for everyone to understand, brings less change and no extra burden on the controller.
I was too trigger happy and did not think about the cases you mentioned above.

Thanks for the suggestion of the approach I will took a stab at your suggestion and here is a draft of the changes to get your suggestion working. I did a quick test and that seem to work fine. There were one too many termination point in call and the has no access to the object of the key. So, instead of reading it took a deferred approach as we already had named return parameters in place on the
Other approach is to use the and fetch the job via key if the is not set to This will do an additional fetch but much simpler in change
This is essentially the same as . The solution is actually very simple, but I didn't have time to implement it Can someone take it? /priority critical-urgent /triage accepted
/assign
Let me open a PR. Do you want to just add the item back to queue or retain the check for that I have in the above ?
just add it back to the queue every time there is a job sync. It's a noop if the item is already there.


Question: if err != nil { return "", err } <del> return types.NodeName(md.Name), nil </del> <ins> return types.NodeName(md.Hostname), nil </ins> } func (i *Instances) AddSSHKeyToAllInstances(user string, keyData []byte) error {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet.

Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).