Question: // Run scheduler. informerFactory.Start(ctx.Done()) <ins> informerFactory.WaitForCacheSync(ctx.Done()) </ins> go sched.Run(ctx) // Send pods to be scheduled.

Which test(s) are flaking: pkg/scheduler unit tests occasionally timeout at 5 minutes Reason for failure: Stats from 10 runs of the package: The average (excluding the run that timed out) is ~40 seconds. The run that timed out took at least 5 minutes before being killed. That smells like a deadlock / failure-to-progress scenario which could be a real bug. Our presubmit CI runs tolerate flakes, retrying 3 times and passing on a single success, which is masking this issue. Observed in which is turning off that toleration and exercising every unit test package 10 times to ensure consistent passes: I can reproduce this by building the unit test package with: and then running repeatedly with : this appears to hang increasing numbers of parallel runs on the deadlock until timeouts are hit: /sig scheduling
the first step is to try individual tests in that package to see if a single test is deadlocking Stressing each test individually with short timeouts (set at at least 5x the normal run time of the test) and seeing if they pass with no timeout failures after 2x the specified timeout: [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [x] [ ] takes ~.1 seconds on average, immediately starts failing with stress test timeout of 10s, fix in [x] [x] [x] [x] takes 15 seconds on average [x] [x] [x]
can consistently reproduce a hang with: I this debugging: and the hang appears to be while waiting for the create calls and resulting events to be produced:
the full goroutine dump at the time of the hang: <details</details
this is also reproducible against the release-1.18 branch
could you help take a look at the problem in ?
/priority important-soon
/assign
PR up for review Thanks a lot for the repro steps TIL
fantastic, that resolves the MultipleProfiles flake:
TestSchedulerWithVolumeBinding looks flaky as well (see )
I see. I went by the title thinking that the other one was already resolved. Let me have a look.

np, good to knock them out one at a time... the symptoms were consistent so I left it as one issue thinking there might be a shared root cause, but that doesn't appear to be the case
my mistake, I got the wrong average runtime for the volume binding test... it actually takes ~16 seconds because we artificially wait for channel timeout for ~2 seconds for each of the subtests. when I stressed it with a timeout of 1m, I got no flakes separately, we could look into improving the runtime of that test by selecting between error/bind/timeout with something like the following, but that is a different cleanup/improvement issue, not a flake fix


Question: # Find all of the built client binaries local platform platforms platforms=($(cd "${LOCAL_OUTPUT_BINPATH}" ; echo */*)) <del> for platform in "${platforms[@]}"; do </del> <ins> for platform in "${platforms[@]}" ; do </ins> local platform_tag=${platform///-} # Replace a "/" for a "-" <del> kube::log::status "Starting tarball: client $platform_tag" </del> <ins> kube::log::status "Building tarball: client $platform_tag" </ins> <del> ( local release_stage="${RELEASE_STAGE}/client/${platform_tag}/kubernetes" rm -rf "${release_stage}" mkdir -p "${release_stage}/client/bin" </del> <ins> local release_stage="${RELEASE_STAGE}/client/${platform_tag}/kubernetes" rm -rf "${release_stage}" mkdir -p "${release_stage}/client/bin" </ins> <del> local client_bins=("${KUBE_CLIENT_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then client_bins=("${KUBE_CLIENT_BINARIES_WIN[@]}") fi </del> <ins> local client_bins=("${KUBE_CLIENT_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then client_bins=("${KUBE_CLIENT_BINARIES_WIN[@]}") fi </ins> <del> # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # KUBE_CLIENT_BINARIES array. cp "${client_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/client/bin/" </del> <ins> # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # KUBE_CLIENT_BINARIES array. cp "${client_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/client/bin/" </ins> <del> kube::release::clean_cruft </del> <ins> kube::release::clean_cruft </ins> <del> local package_name="${RELEASE_DIR}/kubernetes-client-${platform_tag}.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." ) & </del> <ins> local package_name="${RELEASE_DIR}/kubernetes-client-${platform_tag}.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." </ins> done <del> kube::log::status "Waiting on tarballs" wait || { kube::log::error "client tarball creation failed"; exit 1; } </del> } # Package up all of the server binaries

After 6 minutes or so of using 50% my 4-core MBP CPU and chewing up a ton of memory, make release gives up with a timeout.
This is on my master branch (no local changes) after rebasing to head this evening. It was working fine earlier today so seems like the likely culprit. /cc
It's certainly my changes. I'm on a low function device, feel free to revert. This is only an issue for low memory systems, I suspect. :/ On Apr 1, 2015 10:24 PM, "Robert Bailey" wrote:
(This is functioning fine on Jenkins and my desktop, FWIW.) On Apr 1, 2015 10:36 PM, "Zachary Loafman" wrote:


Question: "path" "path/filepath" "strings" <ins> "syscall" </ins> "github.com/golang/glog" "k8s.io/api/core/v1"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: name = name + ".vhd" err = blobClient.PutPageBlob(vhdContainerName, name, vhdSize, tags) if err != nil { <ins> // if container doesn't exist, create one and retry PutPageBlob detail := err.Error() if strings.Contains(detail, errContainerNotFound) { err = blobClient.CreateContainer(vhdContainerName, azs.ContainerAccessTypeContainer) if err == nil { err = blobClient.PutPageBlob(vhdContainerName, name, vhdSize, tags) } } } if err != nil { </ins> return "", "", fmt.Errorf("failed to put page blob: %v", err) } // add VHD signature to the blob h, err := createVHDHeader(uint64(size)) if err != nil {

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: Ansible What happened: PVC keeps pending with a new created storage account What you expected to happen: Storage class can find storage account and then create a dynamic pv for pvc to bind How to reproduce it (as minimally and precisely as possible): a new storage account without create any container a storage class as below a pvc annotating this storage class pvc status is pending Anything else do we need to know: After I create a new container "vhds" in storage account of "qewestus", storage class works and pvc is bound
VHD container is a rerequiste. Will update the doc.
.. I was about to open this exact bug. Shouldn't the provisoner create the vhds container ?
yes, there is a todo for this case (also missing storage account)
Another minor issue is, new volumes take a lot of time (tens of mins) to be ext4 formatted. This causes the pod to timeout and fail upon first use! Any way to make this better
fix is in
Can anyone share how to create the vhds container? For those that don't yet have this fix
Easiest is through the portal, browse to your storage account and create a container called
Thats great thanks for the quick response. The automator in me now wants to see if I can automate the creation of that container through powershell ;)
Does the container have to be of type 'Container' ? IE Full public read access??

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Linux Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: hack/local-up- What happened: storage class pvc annotated with this storage class a pod consume this pvc read/write function with the mounted path pod/pvc the pv status which is "Failed" What you expected to happen: PV should not have "Failed" status before deleted How to reproduce it (as minimally and precisely as possible): Always Anything else do we need to know:
cc
Per message The disk is still being attached. As a result, PV cannot be deleted.
This condition is handled by the PV framework. See However the volume plugin must return which it looks like the Azure volume plugin is not doing.
please help remove label, kubectl can't do anything with this.


Question: // // Enable pods to consume pre-allocated huge pages of varying page sizes HugePages utilfeature.Feature = "HugePages" <ins> // owner @brendandburns // alpha: v1.8 // // Enable nodes to exclude themselves from service load balancers ServiceNodeExclusion utilfeature.Feature = "ServiceNodeExclusion" </ins> ) func init() {

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: func AddOrUpdateLabelOnNode(c *client.Client, nodeName string, labelKey string, labelValue string) { patch := fmt.Sprintf(`{"metadata":{"labels":{"%s":"%s"}}}`, labelKey, labelValue) <del> err := c.Patch(api.MergePatchType).Resource("nodes").Name(nodeName).Body([]byte(patch)).Do().Error() </del> <ins> var err error for attempt := 0; attempt < UpdateRetries; attempt++ { err = c.Patch(api.MergePatchType).Resource("nodes").Name(nodeName).Body([]byte(patch)).Do().Error() if err != nil { if !apierrs.IsConflict(err) { ExpectNoError(err) } else { Logf("Conflict when trying to add a label %v:%v to %v", labelKey, labelValue, nodeName) } } else { break } time.Sleep(100 * time.Millisecond) } </ins> ExpectNoError(err) }

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance] {Kubernetes e2e suite}


Question: rm -rf "${ROLLBACK_BACKUP_DIR}" mkdir -p "${ROLLBACK_BACKUP_DIR}" cp -r "${DATA_DIRECTORY}" "${ROLLBACK_BACKUP_DIR}" <del> rm -rf "${DATA_DIRECTORY}"/member/snap/*.snap </del> echo "Performing etcd3 -> etcd2 rollback" ${ROLLBACK} --data-dir "${DATA_DIRECTORY}" if [ "$?" -ne "0" ]; then

I tried to do a rollback on my own test cluster and got a panic. I will attach the data-dir tarball. The script that performs the migration goes from 2.2.1-2.3.7-3.0.14-2.3.7, and it is this last step that barfs. Script is forthcoming in a PR, but there's a snapshot at
This seems that it's not even about the real data, but about some metadata of the cluster. - any thoughts on it?
Is the data you uploaded the original data before 2.2.1-2.3.7?
yes
The rollback tool couldn't accept any v2 data existed. One way to make it work is to remove snap files:
To clarify, the panic is caused by existing v2 snap files. In the migrate step 2.3.7 -3.0.x, the script didn't remove v2 snap files.
- thanks a lot for looking into it. I will add a step to our migration script to remove the v2 data before rollback. Did it work after removing the data?
send out for review
To clarify - I verified that it fixed the problem reported (rollback works on the data provided in the issue).
The fix in the shell script is good, but the error message in the rollback tool could definitely be improved upon.
says that removing the snap file was not the right way to go.
i marked as milestone 1.5 and marking this as a non-release-blocker. hope this is correct.
Hmm... I tried re-running this on my test data, and now 2.3.7 always panics after the last step, whether I begin with backup data or not. Here is the data before running rollback: after running rollback:
I do not understand this. Does the migration tool still panic? Or the etcd 2.3.7 panics? etcd 2.3.7 should panic if you build the migration tool with 3.0 wal pkg. I believe I sent you the correct migration binary over slack.
Can we close this issue if the rollback tool stop panics? If the migrated data panics etcd 2.3, it is expected. And there is some other issues tracking it. I think we went though them a few weeks ago before thinks giving.

your slack message was lost while I was out of office, but the attached file is still there. Trying it now.
it works with the binary you shared. We still need a way to build that binary, but I will open a separate issue about that.


Question: if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from client to backend: %v", err) } <del> wg.Done() </del> <ins> close(writerComplete) </ins> }() go func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped. What you expected to happen: Kubelet should cleanup the port-forward when it is closed How to reproduce it (as minimally and precisely as possible): Using the following script I can observe an increase in Kubelet memory usage on the node where the pod runs: 300 connections allocates roughly 30mb by kubelet We first noticed this on the node where Tiller (Helm) runs. The node regularly runs out of memory because of the Kubelet memory usage. see also We don't experience the helm issue on all clusters, but I can reproduce the memory leak with the above script on all clusters. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian jessie Kernel (e.g. ): Install tools: Kops 1.8.0 Others:
/sig node In my testing (v1.8.0), I observed create a TCP connection between apiserver and kubelet for each port-forward (and one between apiserver and kubectl). This apiserver<-kubelet TCP connection was not torn down when the corresponding command was killed. I have not tried against a newer apiserver/kubelet, but if someone has one handy, it's easy to just compare output on the apiserver or kubelet host before and after running several commands. Be aware that the CLOSE_WAIT connections can take minutes to expire after the kubectl<-apiserver commands exit, and this is normal and fine - we're interested in the ones that stay in state ESTABLISHED.
I'm not aware of any changes in 1.9 or 1.10 that would have fixed this. Not that we shouldn't fix this issue, but something like would be a good backstop to leaking connections like this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale behaviour is still observable for us on 1.9
On v1.11 I see plenty of and connections that hang around after many minutes.
I some logging to (among many other places) After closing a port-forward from kubectl, I see immediately the reader close on the apiserver, and the writer close on the kubelet. I do not see the other side of the connections close. Maybe 1 on each side when the pod ends. Is there a flag to enable pprof so we can dump some goroutine stacktraces? /sig apimachinery
kubelet stacktrace, the important parts
/sig api-machinery
apiserver stacktrace, the important parts
Lastly, if you restart on one side, the other side does clean up. At least if it's the kubelet. I restarted the kubelet, and apiserver shows Which is the ~600 waiting goroutines.
Based on some conversation with I will look into the approach for this issue. That issue () seems like a similar problem.


Question: <ins> #!/bin/bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. echo "This is NOT a production-ready tool.n IT'S A HACKY, BEST-EFFORT WAY TO "STOP" CREATION OF THE GKE CLUSTER." read -n 1 -p "Are you sure you want to proceed (y/N)?: " decision echo "" if [[ "${decision}" != "y" ]]; then echo "Aborting..." exit 0 fi set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. if [ -f "${KUBE_ROOT}/cluster/env.sh" ]; then source "${KUBE_ROOT}/cluster/env.sh" fi source "${KUBE_ROOT}/cluster/gke/util.sh" STAGING_ENDPOINT="CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER=https://staging-container.sandbox.googleapis.com/" detect-project cluster=$(gcloud container operations list "--project=${PROJECT}" | grep "CREATE_CLUSTER" | grep "RUNNING" || true) if [ -z "${cluster}" ]; then echo "Couldn't find any cluster being created in production environment. Trying staging..." cluster=$(env ${STAGING_ENDPOINT} gcloud container operations list "--project=${PROJECT}" | grep "CREATE_CLUSTER" | grep "RUNNING" || true) fi if [ -z "${cluster}" ]; then echo "No cluster creation in progress found. Aborting." exit 0 fi zone=$(echo "${cluster}" | tr -s "[:blank:]" | cut -f3 -d" ") cluster_name=$(echo "${cluster}" | tr -s "[:blank:]" | cut -f4 -d" ") gcloud="gcloud" if [ "${zone}" == "us-east1-a" ]; then gcloud="env ${STAGING_ENDPOINT} gcloud" fi migs=$(${gcloud} compute instance-groups managed list --project=${PROJECT} --zones=${zone} | grep "gke-${cluster_name}" | cut -f1 -d" ") echo "Managed instance groups for cluster ${cluster_name}: ${migs}" for mig in ${migs}; do echo "Resizing ${mig}..." ${gcloud} compute instance-groups managed resize --project="${PROJECT}" --zone="${zone}" "${mig}" --size=1 done echo "All managed instance groups resized to 1. Cluster creation operation should end soon, and you will be be able to delete the cluster." </ins>

We need a "panic" script for stopping clusters also when they are currently being created. This is needed for the purpose of our performance testing. It should work for at least GCE and GKE.
I think that the first step in GCE and GKE can basically be as simple as: "resize to 0 (or delete)" all MIGs It's far from being perfect but better than nothing.
For GKE that's probably only thing we can do now.
Discussed offline and it seems that for GCE it should be enough to kill the kube-up script and immediately after that start kube-down. Which we already have.
But we still need to for GKE. I'll try to hack something today.


Question: "k8s.io/kubernetes/pkg/client/typed/discovery" "k8s.io/kubernetes/pkg/client/typed/dynamic" "k8s.io/kubernetes/pkg/runtime" <ins> utilerrors "k8s.io/kubernetes/pkg/util/errors" </ins> "k8s.io/kubernetes/pkg/util/sets" "github.com/golang/glog"

I have been having the kube-controller-manager (running in a container) crash quite frequently with this error: I cant fathom what seems to cause it. It seems to happen consistently for a while (CrashLoopBackoff occurs) and then fixes itself after an undetermined amount of time. Kubectl version: Docker version: Networking: Calico (with the calico k8s policy agent) Please do let me know if there is any other information that I can supply. I appreciate I haven't given much to go off of. I am not sure what the steps to reproduce are...
cc
It looks like a bug. I'll take a look. Thanks for reporting it.
Weirdly, the description showed up in the email github sent me but it appears absent on the github issue?
The description was there when I read it but disappeared now.
Well thats very odd! If you have it on your email, could you post it on here please (and I will update the description)? I unfortunately haven't got the error message to hand now...
I took the liberty of just editing the OP.
thanks a lot! Not sure what happened there!
could you paste the returned output at "server-ip:port/api" and "server-ip:port/apis"?
Sure.
Thanks "" has an empty preferred version and that's causing the error you reported. Is "" created via the thirdparty resource api? ()
aha! Good spot! Yes, it was created via that API. This is the Yaml file I used to create it: Here is the output of
Thanks for the confirmation Your configuration is correct. The preferred version is left empty by the thirdparty resource controller, while our discovery client has an assumption that the preferred version won't be empty. I'll patch the discovery client to tolerate an empty preferred version, and will discuss with on if we should make a default preferred version for a thirdparty resource.
Thanks a lot Great job on working it out :)
Sent to fix.
What would be a workaround for this before the release, how can one specify the preferredVersion?
AFAIK, there is no workaround. There is no way to specify the preferredVersion for the thirdPartyResource.

I have a quick workaround to fix this : 1) Change the kubernetes to reflect the following : from : thirdpartyresources=true to : thirdpartyresources=false 2) Then restart 'kubelet' service OR reboot the K8S Master server systemctl restart kubelet 3) Revert back the changes done to kube- from : thirdpartyresources=false to : thirdpartyresources=true 4) Verify whether the above change on the above manifest file has been automatically picked up by the kubernetes API ps aux | grep -i thirdpartyresources 5) Finally you may restart kubelet service systemctl restart kubelet 6) Now verify whether the 'kube-controller-manager' is in a 'Running' state kubectl get pods --all-namespaces | grep -i controller-manager Hope this fix works for you guys as well !!


Question: hk.AddServer(testServer("test2")) hk.AddServer(testServer("test3")) hk.AddServer(testServerError("test-error")) <ins> hk.AddServer(testCobraCommand("test-cobra-command")) </ins> a := strings.Split(args, " ") t.Logf("Running full with args: %q", a)

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: }, }, options: printers.GenerateOptions{}, <del> // Columns: Name, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job2", "0/1", "", "10y"}}}, </del> <ins> // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job2", "Running", "0/1", "", "10y"}}}, </ins> }, // Job with duration. {

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: field: "{.metadata.name}", }, { <ins> name: "random-order-timestamp", obj: &api.PodList{ Items: []api.Pod{ { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(300, 0), }, }, { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(100, 0), }, }, { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(200, 0), }, }, }, }, sort: &api.PodList{ Items: []api.Pod{ { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(100, 0), }, }, { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(200, 0), }, }, { ObjectMeta: api.ObjectMeta{ CreationTimestamp: unversioned.Unix(300, 0), }, }, }, }, field: "{.metadata.creationTimestamp}", }, { </ins> name: "random-order-numbers", obj: &api.ReplicationControllerList{ Items: []api.ReplicationController{

I'd like to be able to sort by a timestamp, e.g. but that fails with an exception: I appreciate that the root of the problem is probably outside of kubernetes, yet I hope you'd agree that sorting by a timestamp is very useful functionality.
Agree with sorting by timestamp.. Just make it clear, is of type: while only these types are allowed to be sorted:


Question: - yujuhong - yliaog labels: <del> - sig/gcp </del> <ins> - area/provider/gcp </ins>

Some owners files attempt to auto apply the no longer existing label, which is now : This is present in 4 owners files: /assign /area oncall-hotlist
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // Remove from oldest to newest (last to first). numToKeep := len(containers) - toRemove for i := numToKeep; i < len(containers); i++ { <del> err := cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: containers[i].id}) </del> <ins> err := cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: containers[i].id, RemoveVolumes: true}) </ins> if err != nil { glog.Warningf("Failed to remove dead container %q: %v", containers[i].name, err) }

I'm conducting PoC about distributed build using Kubernetes. Jenkins master distribute jenkins slave pods through Kubernetes API Server. When build is done, jenkins slave pod is also deleted by kubernetes API Server. But, kubernetes didn't delete container. (Pod is deleted well) After searching, I found kubelet’s containerGC option. So, I adjusted below options to kubelet. --minimum-container-ttl-duration=0s --maximum-dead-containers=0 After adjusting both 2 options, container was deleted, too. But, kubernetes containerGC didn’t delete host volume path. (/var/lib/docker/vfs/dir/..) Due to aufs’s low performance, I used data volume of host instead of container inside. So, data volume was accumulated and host's inodes was fulled, too. Is there any solution to clean the containers wreckage? My environment was.. Docker: v1.6.2 Kubernetes: v0.18.2 (container based multi clustering)
are you using the host volumes you created yourself? If this is the case, kubelet does not clean the volumes since they are external to kubelet. If not, kubelet should clean the volumes before container GC kicks in.
No.. volumes were created by kubenetes API server. LGTM. Will it be adjusted next release?
You said "I used data volume of host instead of container inside" which says you used a hostPath mount - correct? That is explicitly NOT managed by kubernetes - you've gone to the host, you get to manage it. But you also said "Due to aufs’s low performance" - an emptyDir volume is NOT stored on aufs, but it IS managed with the pod. , gghonor wrote:
The issue as i understand it is dangling volumes, that is volumes that are created via a VOLUME directive in Dockerfile with no corresponding kubernetes volume in spec which therefore has no host path & is not cleaned up by the kubelet. These volumes reside in /var/lib/docker/vfs/. This waste of disk space has the potential to cause out of space errors on the node & therefore make the node unstable.
As the volume dir you mentioned is I assume you didn't specify any volume in your Kubernetes pod spec but did use the directive in your Dockerfile?

As mentions I guess this isn't a Kubernetes managed volume as that would be in the Kubernetes volume dir, right? That dir is used by Docker for volumes specified in Dockerfiles or a docker container that is run with with no specified host path. The latter doesn't happen with Kubernetes managed containers AFAIK.
This makes sense. Thanks for the explanation :) I see the point of removing the dangling volumes. Even though kubelet doesn't manage those volumes, it doesn't hurt to remove them.
Do we have a full understanding of their lifetime, though? We don't control them, I don't see how we can know when to delete them. Docker's volume implementation is just too under-defined to actually be useful. , Yu-Ju Hong wrote:


Question: "+someVersionTag:version=1.5,something", }, wantValue: nil, <del> wantErr:   true, // TODO: Today it is klog.Fatal, check how to capture it },*/ </del> <ins> wantFatal: true, }, </ins> { name:    "wrong tag major value should return an error", tagName: "someVersionTag:version",

The commands under will benefit of a batter unit test coverage as seen in Originally posted by in
/sig api-machinery /assign
GitHub didn't allow me to assign the following users: damiansawicki. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Hi I would like to work on this issue. Could you tell me how many test files are needed per PR under this issue?
Hi the issue is already assigned to me. Would you consider contributing to other issues to avoid duplication?
Hi do you mind if I work on this issue along with you? I would only like to write tests on limited number of files, if that's okay with you...
Hi thanks for your engagement! If you also want to work on this particular issue, we should coordinate. Are you working on something other than ?
Hey currently I'm thinking of shifting to Do let me know if you are already working on that particular directory. Also I could assist you in any of your work on this issue as I'm not preoccupied at the moment.


Question: file=${generated_files[$1]//.generated.go/.go} deps=$(go list -f "{{.Deps}}" ${file} | tr "[" " " | tr "]" " ") candidate=$(readlinkdashf "${generated_files[$2]//.generated.go/.go}") <del> result=false </del> for dep in ${deps}; do <del> if [[ ${candidate} = *${dep} ]]; then result=true </del> <ins> # Only look at dependencies within the kubernetes tree-- otherwise the "io" # package matches the end of one of our directories. if [[ ${dep} = "k8s.io"* ]]; then if [[ ${candidate} = *${dep} ]]; then echo true return fi </ins> fi done <del> echo ${result} </del> <ins> echo false </ins> } function tsort {

See, e.g. This seems to be blocking multiple PRs from even being tested.
Looking.
I think I figured it out. The order of generation matters. The verify script subtly breaks that by renaming and checking the files one at a time. The update script deletes everything and then recreates them.
I have a fix. Will send a PR shortly.


Question: createLocalPVCsPVs(config, []*localTestVolume{testVol}, immediateMode) pod, err := createLocalPod(config, testVol, nil) Expect(err).To(HaveOccurred()) <del> checkPodEvents(config, pod.Name, ep) </del> <ins> err = framework.WaitTimeoutForPodRunningInNamespace(config.client, pod.Name, pod.Namespace, framework.PodStartShortTimeout) Expect(err).To(HaveOccurred()) </ins> cleanupLocalPVCsPVs(config, []*localTestVolume{testVol}) })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: deps = [ "//pkg/apis/core:go_default_library", "//pkg/probe:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/internalversion:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/fields:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/diff:go_default_library", "//staging/src/k8s.io/apiserver/pkg/endpoints/request:go_default_library", ],

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: tried to get listing of all componentstatuses using a labelSelector query selector. I am hitting the apiServer directly. GET /api/v1/componentstatuses What you expected to happen: GET /api/v1/componentstatuses with labelSelector should limit responses to only componentstatuses with the specified label How to reproduce it (as minimally and precisely as possible): run GET /api/v1/componentstatuses with and without a labelSelector against the apiServer. Anything else we need to know?: specifically my label selector is in the form . and i can confirm everything works as expected when i hit GET /api/v1/pods Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6+IKS", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-09T13:26:51Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: IBM OS (e.g: ): Mac Kernel (e.g. ): Darwin Kernel Version 18.5.0 Install tools: Using javascript npm request to talk to api server Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
i assume its because you cant add a label to componentstatus, but they still shouldnt return if i specify a labelSelector
that's indeed a bug i presume. but what use-case makes you label-querying componentstatus?
we have a bot to periodically run a query that finds all known endpoints on the ApiServer. Then for each of the found endpoints, find all resources with our label defined. sort of an inventory tracking system.
in a similar vein, the endpoint doesnt seem to respect
/priority backlog /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /priority backlog /help Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will take a look at this unless no objections
thank you, yours!
a PR for the ComponentStatus API, could you pls kindly take a look? I will look into the metrics API probably over the weekend. Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: mounter:  &mount.SafeFormatAndMount{Interface: host.GetMounter(), Runner: exec.New()}, }, nil } <ins> func volumeSpecToUnmounter(mounter mount.Interface) *fcDiskUnmounter { return &fcDiskUnmounter{ fcDisk: &fcDisk{ io: &osIOHandler{}, }, mounter: mounter, } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When user uses fc volume plugin, device file on operating system such as /dev/sdX remains after deleting a Pod. After deleting the Pod, PVC and PV, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up. The photon volume plugin already has this functionality in the removeFromScsiSubsystem(). Also the gce volume plugin has similar functionality in the udevadmChangeToDrive() too. photon: echo 1 /sys/block/<dev name/device/delete gce: udevadm trigger --action=change --property-match=DEVNAME=%s Also os-brick library in the OpenStack Cinder component uses same way with photon driver. What you expected to happen: The device file on operating system such as /dev/sdX will be cleaned up after deleting PVs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: KVM OS (e.g. from /etc/os-release): Fedora 25 Kernel (e.g. ):Install tools: local-up- Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
I'll work on this.
Reiterating the mentions to trigger a notification: <detailsIn response to : I'll work on this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: docker build --pull -t etcd-integration-test $(TEMP_DIR)_integration_test integration-test: <del> docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes -e GOARCH=$(ARCH) etcd-integration-test  </del> <ins> docker run --interactive -v $(shell pwd)/../../../:/go/src/k8s.io/kubernetes$(DOCKER_VOL_OPTS) -e GOARCH=$(ARCH) etcd-integration-test  </ins> /bin/bash -c "CGO_ENABLED=0 go test -tags=integration k8s.io/kubernetes/cluster/images/etcd/migrate -args -v 10 -logtostderr true" integration-build-test: build-integration-test-image integration-test

Part of What happened: Attempt to run as a non-root user Get What you expected to happen: It runs How to reproduce it (as minimally and precisely as possible): /sig release /sig cluster-lifecycle /area security And as part of kubeadm v1.16 milestone: /milestone v1.16 /priority critical-urgent
/assign randomvariable /lifecycle active
/priority important-soon


Question: return server.Run(stop) } <ins> func (o *Options) writeConfigFile() error { var encoder runtime.Encoder mediaTypes := o.codecs.SupportedMediaTypes() for _, info := range mediaTypes { if info.MediaType == "application/yaml" { encoder = info.Serializer break } } if encoder == nil { return errors.New("unable to locate yaml encoder") } encoder = json.NewYAMLSerializer(json.DefaultMetaFactory, o.scheme, o.scheme) encoder = o.codecs.EncoderForVersion(encoder, componentconfigv1alpha1.SchemeGroupVersion) configFile, err := os.Create(o.WriteConfigTo) if err != nil { return err } defer configFile.Close() if err := encoder.Encode(o.config, configFile); err != nil { return err } glog.Infof("Wrote configuration to: %sn", o.WriteConfigTo) return nil } </ins> // NewSchedulerCommand creates a *cobra.Command object with default parameters func NewSchedulerCommand() *cobra.Command { opts, err := NewOptions()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We face But there is no way to generate default config file How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version 1.9.2
I'll take a look it /sig scheduling
/assign
Hi, same issue with controller-manager
Any updates here? , and don't have this flag. Is it even will be supported?
/cc
if not set, component should be use default value; why we'd like to dump all default config?
Because the documentation on the config file format is either nonexistent or difficult to find. If the program itself can generate a template, that fulfills (most of) the need for documentation of the format.
+1 for this, from my expirence, add an option to write the default config is really helpful.
/assign
also in 1.10
Deprecating all flags, prompting folks to use config files in advance is great, except when there's no documentation nor example anywhere. That means that we can't actually migrate to the new settings, and I can't seem to find clear alpha/beta/stable status for the new config file spec, nor timeline for the removal. It is also pretty confusing that the scheduler expects us to use a config file, but the controller-manager does not?
This issue indeed applies to kube-controller-manager. And there should already had deprecation policy.
According to , the flag will be available in . Since I can't find any reference online to an example config file, perhaps one could be provided or linked to here?
will you be able to pick my patch and generate a config file? I do not get my devbox handy so can only post here until Monday.


Question: // It's safer to assume that it's not a mount point. return true, nil } <del> strOut := strings.TrimSuffix(string(out), "n") </del> <ins> mountTarget := strings.Split(string(out), " ")[0] mountTarget = strings.TrimSuffix(mountTarget, "n") glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v:", file, mountTarget) </ins> <del> glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v", file, strOut) if strOut == file { </del> <ins> if mountTarget == file { glog.V(5).Infof("IsLikelyNotMountPoint: %s is a mount point", file) </ins> return false, nil } <ins> glog.V(5).Infof("IsLikelyNotMountPoint: %s is not a mount point", file) </ins> return true, nil }

Kubelet (v1.2.4) can consume a lot cycles and when it keeps trying to tear orphaned volumes down:
See also cc
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
/cc

I'm running Kubernetes v1.2.3 using the single-node instructions here: with Docker v1.10. Everything worked as expected for several hours, but now pods are not being scheduled or removed. I've attempted to update a deployment several times over the last hour. Here is for that deployment: The ContainerCreating pod's last event from is: When I delete a pod, the scheduler does seem to start the new pod almost immediately, and the kubelet logs: The kubelet then begins logging a readiness check failure every 10s: but I can't see any evidence that it actually tries to run the container. Docker doesn't emit any logs. The Docker daemon is responsive, and the image is definitely present. When I run the image using it works fine. Anything I can check? Thanks!
At a glance, looks like some bug in state transitions at the kubelet.
is there any message in the kubelet log related to image pulling? Without further information, it seems like kubelet could be stuck at pulling the images. It'd be helpful to see if there are related information.
Can you run docker ps -a to check if docker failed to create / run containers? Could you please upload related Kubelet log somewhere so that we can access. Based on the initial description, it looks like the image is successfully pulled.
I was just making a wild guess because image pulling is usually where the pod workers get stuck and become unresponsive for a long time :) From the description, the "cluster" was working for a few hours, but the pods were newly created.
and , does not show any attempted containers of the relevant image. Here are the logs of the kubelet. I d the existing pod in the deployment, which was in the state; and a new pod was immediately scheduled. These logs cover those events. The Readiness probes continue at 10s intervals. I sent you the full logs of the kubelet in a Slack pm.
just an update, the kubelet is still in the same state: Some context that may be helpful is this happened after I had been experimenting with deploying an NFS server using the current Kubernetes example, , and . I changed the replication controller at to a deployment, and updated it several times.

I was changing the code inside the container, so some of those attempts may have contained incorrect NFS configuration and/or failed to clean up completely before the container exited. I haven't seen this particular kubelet state before. I have seen slow pod state changes, , but have never seen them halt completely. This state is also different because the kubelet is not logging anything. Previously when I have seen slow pod state changes, it has logged repeatedly about orphaned volumes. Is it plausible that this issue has something to do with the state of the kernel after my experimentation with NFS?
/cc
If this is related to stuck image pulling, what you'll see in the docker daemon's log is something like this: If pulling is stuck, this is all that you'll see. If pulling isn't stuck, then you should see something like the following: And in the kubelet's log, this is the last entry you should expect to see for a pod that is stuck Pending / ContainerCreating if pulling is indeed stuck:
the image is definitely present on the node. I've verified by hand: and also the kubelet doesn't log about creating an image. The logs here: are all it says after a pod is scheduled. Note also that several pods have been in the 'Terminating' state for three days.
can you ssh to the node and curl the kubelet port to get a stack trace? e.g.
Stack trace attached.
goroutine 192 looks
What does it mean?
I think it's trying to mount the NFS directory. You said you had an incorrect configuration at one point?
There are several goroutines in this state
It's safe to assume I had an incorrect configuration at one point, yeah.
Wrong IP address for the NFS server, or ? , Anand Patil wrote:
To clarify, that deployment doesn't consume any NFS persistent volumes, and none of my versions did. It consumes a host volume and runs an NFS server. The pod spec is attached.
So it can't be hanging trying to mount an NFS volume, it's trying to create an NFS volume.
By "incorrect configuration", I meant that my version of this: could have been wrong.
Oh ok. Here's one of several hung pod workers: I don't know that this pod worker is necessarily associated with the nfs server pod. Regardless, it's trying to run And hanging.

You probably need to get some help from when they have time.
OK, I appreciate your attention to this issue, I look forward to hearing from them.
do you have installed on the Docker host?
xref
Hi unfortunately the machine where this issue was observed has been shut down. I saw something similar since then, though, and have a hunch about what was going on. A running pod had an NFS volume source, then the NFS server was shut down. The result was that there were some folders in in which recursive walkers like and would hang as described here: . I can try to create a simple repro case, if that would be helpful.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.

Kubelet continuously throws lots of errors: I'm starting the cluster this way:
To reproduce this behavior I cleaned up everything related to kubernetes (containers, /var/lib/kubernetes contents, etc), started hyperkube, then started a bunch of containers - everything is good up to this point. Then I delete a pod with (kubectl delete pod frontend), and this happens:
More observations. Here is my default token: Here is error message I get: This path is changing all the time: (Note the subdirectory names are different) Pod deleteion code was recently touched by authentication tokens (whose directories can't be removed) by Any ideas what it could be? I would ignore this problem if it didn't consume 15% of CPU in a very tight orphan removal loop.
Those tokens are just secrets as far as the kubelet is concerned. Not sure how secret volume cleanup differs in hyperkube. cc
We have same problem on nodes with CoreOS alpha 991.0.0 and hyperkube 1.2 a lot of messages like:
same issue here.
Same on Ubuntu 14.04, hyperkube v1.2.3, docker 1.9.1 with aufs backend, kernel 3.13.0-48-generic. I also see pod state transitions taking a long time; they hang in "Pending" and "Terminating" for minutes. Does anyone else see that in conjunction with this issue?
For context, this issue doesn't show up immediately in my hyperkube setup. It turns up after a few days. During that time, I update 6 or 7 deployments simultaneously, several times per day.
Sorry I missed the ping on this. Assigned to self, will try to triage by the end of the week.
cc
got the same problem on centos
I am wondering if this depends on Docker version, and not so much the graph driver.
xref
Same problem here, any ideas?
could you please re-triage this bug. I'm not sure "team/ux" is the right tag for it - it doesn't have anything common with UX. It looks like a problem with some low-level container machinery. And given the number of +1's, maybe raise the priority? Thanks.
What problem does this cause for you?
!

The error msg will be a confuse to log/alert monitor infrastructures and really hope it can be fixed.
As I wrote before: I have a 8-core laptop that can work ~ 4 hours on battery (enough for a mid-range flight). When I run Kubernetes cluster via hyperkube, it loads 15% of my CPU resources (more than 1 core!) in a tight loop draining my battery less than in hour.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
After setting up a hypekube and digging the logs, I think the following happened and caused the issue. Hyperkube automatically creates a few pods for dns, dashboard, proxy services etc. But there is something wrong when setting up and mounting the volumes. The following shows some logs from one such pod. (1) 21:20:03. ] SyncLoop (ADD, "api"): "kubernetes-dashboard-fnivxkube-system(-29d0-11e6-bafc-)" (2) 21:20:03. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-]  These two steps show that volume is mounted (3) 21:20:07. ] IsLikelyNotMountPoint findmnt output for path /var/lib/kubelet/pods/-29d0-11e6-bafc- /var/lib/kubelet/pods/-29d0-11e6-bafc-  Very strange thing here is when SychPod check the mount path, the last letter "7" of the path is missing. Now the step (4) show that the volume is mounted again (4) 21:20:07. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-] From this point, each time SyncPod check the volume, it mounts again the again which is proved by the findmnt output in the log. After a few hours running, there are over hundreds of mounts.

I am working on finding the reason why the last letter of the returned mount point is missing. I manually tested the findmnt command inside of the container and it returned the correct result. There is also an issue for umount. The umount code will first move the mount dir to a temporary dir, but failed with "device or resource busy" error. I also tried to mount tmpfs in the container with command mount -t tmpfs tmpfs userdir Then tried to mv userdir also failed with the same error. Will continue working on solving this issue. Please let me know if you any comments
Excellent debugging Looking forward to seeing what the root cause is for both issues.
Some updates for this bug issue one that path name missing the last character: I tried to run the "findmnt" command directly on terminal, the output is fine. But when executing the command though exec.command.Output(), the target path name's last character always misses. I tried out different findmnt options, it does not work. Only if the mounting path name is relatively short (looks like under 90~100 characters), there is no missing character. Also no matter how long the path is, only the last character is missing. Our current code in function nsenter_mount.IsLikelyNotMountPoint() uses exact string match to detect whether the path is already mounted or not, which is not robust. We can change to substring match instead. umount issue. The tearDown also invokes IsLikelyNotMountPoint(). If it returns error message, GetMountMedium() function will regard the mount medium as default, but instead it should be memory. Then teaddown will invokes tearDownDefault and tries to rename directories and delete them, which in turn results in "Device or resource busy" error. Also it does not handle the case if path is mounted multiple times (this case should not happen in normal situation) I will try to modify the IsLikeLyNotMountPoint() to make it more robust. Please let me know if you have any comments or suggestions. Thanks!
PR and PR both modified the behavior of to fix issues and evidently introduced different issues. So watch out for regressions in any potential fix. CC Also, have you observed the missing character issue in a non-hyperkube environment, or is there something unique about running locally inside a docker container that causes this to happen?

I see isLikelyNotMountPoint() function is implmented in two places: and My understanding is nsenterMounter is only used when kubelet is running inside of container. nscentermount uses findmnt command result to check the mount point. In , it uses a different approach: It checks whether the directory has a different device as parent, then it is a mountpoint I also wrote a small go program to test the "findmnt" command in my local environment, it also truncate the last letter if the mounting path is relatively long. So this "findmnt" behavior is not related to hyperkube environment I think.
Thanks for the clarification
Thanks for debugging this. May I set the milestone, as I think we should fix this before release?
Sure. THanks! Jing , Lucas Käldström wrote:
Please note that findmnt command does not exist on docker host vm for virtualbox/windows setup (i.e. 'boot2docker'). I'm pretty sure this will be the case for Mac too. The errors that I get are: ] Failed findmnt command for path /var/lib/kubelet/pods/-31ba-11e6-bdbe- exit status 1 and these don't seem to get cleaned up
This problem is still there. Right now I'm observing it on my GKE instances (managed VM, Kubernetes 1.3.5): I noticed it after CPU load jumped from typical 15% to crazy 100% on both my nodes: !
Thank you for providing this information. After checking the log offline, the problem should be fixed in v1.3.6. Please upgrade to v1.3.6 and let us know if you have any problem. Thanks!


Question: path: /var/lib/docker/containers - name: config-volume configMap: <del> name: fluentd-es-config-v0.1.5 </del> <ins> name: fluentd-es-config-v0.1.6 </ins>

Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective.

I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue.

If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together?

The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", </ins> "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library", "//vendor/github.com/prometheus/client_model/go:go_default_library", ],

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: // RangeSize returns the size of a range in valid addresses. func RangeSize(subnet *net.IPNet) int64 { ones, bits := subnet.Mask.Size() <del> if bits == 32 && (bits-ones) >= 31 || bits == 128 && (bits-ones) >= 63 { </del> <ins> if bits == 32 && (bits-ones) >= 31 || bits == 128 && (bits-ones) >= 127 { </ins> return 0 } <del> max := int64(1) << uint(bits-ones) return max </del> <ins> // For IPv6, the max size will be limited to 65536 // This is due to the allocator keeping track of all the // allocated IP's in a bitmap. This will keep the size of // the bitmap to 64k. if bits == 128 && (bits-ones) >= 16 { return int64(1) << uint(16) } else { return int64(1) << uint(bits-ones) } </ins> } // GetIndexedIP returns a net.IP that is subnet.IP + index in the contiguous IP space.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating an IPv6 cluster, the --pod-network-cidr needs to be a /66, it cannot be for a larger network (smaller value) What you expected to happen: Should be able to use values smaller then /66, /64 or a /48 How to reproduce it (as minimally and precisely as possible): Create a cluster with Kubeadm, and set the podSubnet as below networking: podSubnet: "fd00:1:2:3::/64" Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /area ipv6 /sig network
/assign rpothier


Question: "k8s.io/kubernetes/pkg/apis/rbac" "k8s.io/kubernetes/pkg/client/clientset_generated/clientset/fake" "k8s.io/kubernetes/pkg/client/restclient" <del> openapigen "k8s.io/kubernetes/pkg/generated/openapi" </del> "k8s.io/kubernetes/pkg/genericapiserver" genericapirequest "k8s.io/kubernetes/pkg/genericapiserver/api/request" kubeletclient "k8s.io/kubernetes/pkg/kubelet/client"

Haven't seen this one before and I am not sure it has any relation with the that hit it. https://k8s-
From what I can gather, it is being killed due to taking over 5 minutes which is roughly how long it takes (~300s) on successful unit builds.
bumping priority. I'm seeing this pretty frequently this morning. I see OpenAPI and associate you. You know something about this one?
Yes, I wrote this test. It was failing (timing out) for quite some time on jenkins but passing quite fast locally. I've disabled it but after a while it started passing and I assumed it was a configuration change on test machines. I will spend some time on it again to see if I can find the root cause and will disable it otherwise. This test is passing locally in about 15 second. Do you have any idea why it is timing out (or passing near to 300s) on jenkins machines? where do you see the successful test stats?
I see what you do. Locally it takes 25s, with bots it is 300s. My guess is that we are constrained by CPU and all the encode/decode is causing strain either on CPU or GC. The bots fail about 75% for me so I just keep retrying until success. Just had 2 this morning pass at 296s.
Curiously, it's taking about 20 seconds with Bazel. this might be worth looking at.
I think we run tests in Jenkins with , which might explain the slowness?
Nevermind, we hit it in a bazel run: https://k8s-
-race does kill the performance, 25s -166s which is on par with bazel.
is it possible to disable -race for this test? and should we do that if it is possible?
Now blocking the submit queue.
I think I am missing something here. I've checked some of the attached PRs. Links to the pkg/master folder in master branch has the change (new file called but non of the recent PRs I checked has the file in their branch: That means they didn't sync with master? What am I missing here?
OK. I see, the tests are patching the change of the PR into the master, so everybody should get that if they retest, but it look like my test is not blocking submit queue. there should be another issue.


Question: }) } } <ins> func TestPatchPodStatus(t *testing.T) { tests := []struct { name           string pod            v1.Pod statusToUpdate v1.PodStatus }{ { name: "Should update pod conditions successfully", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns", Name:      "pod1", }, Spec: v1.PodSpec{ ImagePullSecrets: []v1.LocalObjectReference{{Name: "foo"}}, }, }, statusToUpdate: v1.PodStatus{ Conditions: []v1.PodCondition{ { Type:   v1.PodScheduled, Status: v1.ConditionFalse, }, }, }, }, { // ref: #101697, #94626 - ImagePullSecrets are allowed to have empty secret names // which would fail the 2-way merge patch generation on Pod patches // due to the mergeKey being the name field name: "Should update pod conditions successfully on a pod Spec with secrets with empty name", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns", Name:      "pod2", }, Spec: v1.PodSpec{ // this will serialize to imagePullSecrets:[{}] ImagePullSecrets: make([]v1.LocalObjectReference, 1), }, }, statusToUpdate: v1.PodStatus{ Conditions: []v1.PodCondition{ { Type:   v1.PodScheduled, Status: v1.ConditionFalse, }, }, }, }, } client := clientsetfake.NewSimpleClientset() for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { _, err := client.CoreV1().Pods(tc.pod.Namespace).Create(context.TODO(), &tc.pod, metav1.CreateOptions{}) if err != nil { t.Fatal(err) } err = PatchPodStatus(client, &tc.pod, &tc.statusToUpdate) if err != nil { t.Fatal(err) } retrievedPod, err := client.CoreV1().Pods(tc.pod.Namespace).Get(context.TODO(), tc.pod.Name, metav1.GetOptions{}) if err != nil { t.Fatal(err) } if diff := cmp.Diff(tc.statusToUpdate, retrievedPod.Status); diff != "" { t.Errorf("unexpected pod status (-want,+got):n%s", diff) } }) } } </ins>

I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted


Question: setup-addon-manifests "addons" "0-dns/nodelocaldns" local -r localdns_file="${dst_dir}/0-dns/nodelocaldns/nodelocaldns.yaml" setup-addon-custom-yaml "addons" "0-dns/nodelocaldns" "nodelocaldns.yaml" "${CUSTOM_NODELOCAL_DNS_YAML:-}" <del> # Replace the sed configurations with variable values. sed -i -e "s/_.*_DNS__DOMAIN__/${DNS_DOMAIN}/g" "${localdns_file}" sed -i -e "s/_.*_DNS__SERVER__/${DNS_SERVER_IP}/g" "${localdns_file}" sed -i -e "s/_.*_LOCAL__DNS__/${LOCAL_DNS_IP}/g" "${localdns_file}" </del> <ins> # eventually all the __PILLAR__ stuff will be gone, but theyre still in nodelocaldns for backward compat. sed -i -e "s/__PILLAR__DNS__DOMAIN__/${DNS_DOMAIN}/g" "${localdns_file}" sed -i -e "s/__PILLAR__DNS__SERVER__/${DNS_SERVER_IP}/g" "${localdns_file}" sed -i -e "s/__PILLAR__LOCAL__DNS__/${LOCAL_DNS_IP}/g" "${localdns_file}" </ins> } # Sets up the manifests of netd for k8s addons.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gci-gce-serial-kube-dns-nodecache Which test(s) are failing: N/A Since when has it been failing: 10/13 Testgrid link: https://k8s- Reason for failure: Cluster failed to start: Anything else we need to know: /cc
So, is a failing job, we can see that there is a error around binding to a specific IP address. In passing jobs, the initial core file isn't printed out () so not sure how to do an apples to apples comparison of what is broken here yet.
/kind bug I have been looking at this offline with and The only difference between a failing and a passing job is the arguments to the bind plugin. Failing Job: Link - Image - Config: Passing Job: Link - Image: Config:
Issue is reproducible when spinning up a cluster via kubetest using:
changes to lines 907-910 in hack/local-up- and lines 2467 to 2469 in configure- seem like the issue.
What we see is substitution works correctly for line , but for line , it replaces "PILLARLOCALDNS" with empty string.
/assign Can you investigate?
Yeah lookin into it
Regex is matching the entire line :). Should be easy fix . Will verify in hack local up and issue PR shortly
/sig network /triage accepted


Question: "math" "time" <del> meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" </del> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/instrumentation"

There's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time


Question: } } <ins> // DeleteNamespace can be used to delete a namespace. Additionally it can be used to // dump namespace information so as it can be used as an alternative of framework // deleting the namespace towards the end. func (f *Framework) DeleteNamespace(name string) { defer func() { err := f.ClientSet.CoreV1().Namespaces().Delete(context.TODO(), name, metav1.DeleteOptions{}) if err != nil && !apierrors.IsNotFound(err) { Logf("error deleting namespace %s: %v", name, err) return } err = WaitForNamespacesDeleted(f.ClientSet, []string{name}, DefaultNamespaceDeletionTimeout) if err != nil { Logf("error deleting namespace %s: %v", name, err) return } // remove deleted namespace from namespacesToDelete map for i, ns := range f.namespacesToDelete { if ns == nil { continue } if ns.Name == name { f.namespacesToDelete = append(f.namespacesToDelete[:i], f.namespacesToDelete[i+1:]...) } } }() // if current test failed then we should dump namespace information if !f.SkipNamespaceCreation && ginkgo.CurrentGinkgoTestDescription().Failed && TestContext.DumpLogsOnFailure { DumpAllNamespaceInfo(f.ClientSet, name) } } </ins> // CreateNamespace creates a namespace for e2e testing. func (f *Framework) CreateNamespace(baseName string, labels map[string]string) (*v1.Namespace, error) { createTestingNS := TestContext.CreateTestingNS

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up.

But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign


Question: } } <del> pod.Status.Conditions = append(pod.Status.Conditions, api.PodCondition{ </del> <ins> podutil.UpdatePodCondition(&pod.Status, &api.PodCondition{ </ins> Type:    api.PodScheduled, Status:  api.ConditionFalse, Reason:  apiv1.PodReasonSchedulingGated,

When a scheduling gated pod is ungated, the condition's is consistently null until the pod is scheduled: Pods without a scheduling gate have the same condition but is populated: Submit a pod with a scheduling gate. Observe that after removal of the gate, the pod condition. I uncovered this while using Kueue and the ProvisioningRequest feature. Since this is reproducible with just the scheduling gate, I think k/k is the right project for the issue. <details</details<detailsGCP </details<details</details<details</details<details</details<details</details
/cc
/sig scheduling
When the pod is gated, what's the condition of the Pod? Is it empty?
The gated pod has just this condition. The transition time is also null, but is that expected since it is the first condition?
/triage accepted
The bug is here: It looks like at this point, the ObjectMeta.CreationTimestamp is already populated. We should copy that field.
/assign
Are you still on this, we meet the same problem.
It's been over a month, I assume you might no longer work on this? We encounter the same bug and hope this bug can be fixed as soon as possible(hopefully next release). Since there is no further update on this issue, I am proposing a protential fix on now, can you assign this issue to me and review the code once you are available? Thanks.
/unassign ahaysx /assign royliang
GitHub didn't allow me to assign the following users: royliang. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /unassign ahaysx /assign royliang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry I have been on vacation until today. Thanks for the fix
are you still working on this?
I’ve submitted the PR, but it seems lack of reviewer moving this forward.