Question: continue } // A write lock is acquired to make sure the cleanUp routine doesn't delete any VM's created by ongoing PVC requests. <del> defer cleanUpDummyVMLock.Lock() err = diskmanagers.CleanUpDummyVMs(ctx, vmFolder, dc) if err != nil { glog.V(4).Infof("Unable to clean up dummy VM's in the kubernetes cluster: %q. err: %+v", vs.cfg.Workspace.Folder, err) </del> <ins> cleanUpDummyVMs := func() { cleanUpDummyVMLock.Lock() defer cleanUpDummyVMLock.Unlock() err = diskmanagers.CleanUpDummyVMs(ctx, vmFolder, dc) if err != nil { glog.V(4).Infof("Unable to clean up dummy VM's in the kubernetes cluster: %q. err: %+v", vs.cfg.Workspace.Folder, err) } </ins> } <ins> cleanUpDummyVMs() </ins> } }

/kind bug /sig release call will never be execute.
/assign
Maybe just cleanUpDummyVMLock.Lock() is you want? The is in infinite loop, and will never execute. I hope to get your reply to help fix it.
yes we need to correct this as below in the routine.


Question: echo "SALT_MASTER='${MASTER_INTERNAL_IP}'" echo "MINION_IP_RANGE='${MINION_IP_RANGES[$i]}'" echo "DOCKER_OPTS='${EXTRA_DOCKER_OPTS:-}'" <ins> grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/common.sh" </ins> grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/format-disks.sh" grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/salt-minion.sh" ) > "${KUBE_TEMP}/minion-start-${i}.sh"

I saw this error in on the master after a failed cluster start: I'm looking into this, but I want to create a tracking issue as I think some other people may be hitting this.
It's trying to bring up and failing: The weird thing is that neither that file nor the file have been modified that recently:
Yes, to me it looks like salt just isn't finding the plugin. The last change to _states did add , but I've seen this succeed recently as well (http://kubernetes-). I'm going to try this a few times, to try to figure out whether it is failing every time or not. My guess is that it is failing semi-randomly, so it isn't related to the changes since last-fail.
Aha... this is logged at debug level from I'm guessing it's a salt version mismatch... investigating.
I guess AWS needs to lock salt versions like GCE does:
Confirmed that this is because of upstream salt breakage: - deprecated (?) Not sure what we should do here. I can lock to the same version as GCE, but if we find a bug in that version of salt then we can't update without breaking this script.
We don't write bugs. ;)
Can you just put this in an AWS-specific block? It seems like that should work.
(And by "this" I mean "future Salt variable gunk")
I think I'm going to try to lock AWS to the same version as GCE, and then think more about a long-term plan once AWS is working again!
from the looks of things I think vagrant will have the same problem (?) (Note that I am in the process of developing / testing the branch above, so I don't think you can just copy-and-paste the fix - yet!)
As a short term fix, using 2015.05 (ubuntu trusty salt ppa), was to change from: import as ipaddr to: import as ipaddr this actually seems to be working. might be deprecated. but not in 2015.05. why is there a containerbridge custom state at all btw? why not use network.managed and
- I'm not sure we can just change the import, because I think we have to be compatible with GCE's version of salt also.

I'm not sure of why we did it this way vs other ways of doing things in salt
Does this mean that the official k8s distribution is broken on AWS, BTW? When is the next point release (and where are these discussed)?
We cherry-pick into point releases as necessary, and it's been pretty informal and usually initiated by us (Google). I'd actually be happy to have a discussion about how to initiate it externally (we're going to need a policy for requesting such things). cc
FWIW, policy aside, I'm happy to cut a point release for this once you think is tested well enough.
That would be great, thanks. I'll be testing it pretty hard today, I think. It has occurred to me that other clouds may also be broken. By grepping, it looks like vagrant and vsphere use I am utterly unfamiliar with the vsphere implementation, and I think vagrant sidesteps the issue because on Redhat is skips the call to It might be good to get confirmation that vagrant hasn't fallen in to a different trap though! Let me test it out today (unless people start filing lots of issues, in which case I think we roll it out more urgently...)
I will test out the Vagrant setup today, but I think we should be good because we do not use the same networking setup.
File a bug for a 0.17.1 to get this cherry-picked and assign it to me, as a reminder?
Do you know the status of 17.1? I'm seeing the same problems trying to AWS working, even tried an older release (e.g. v15), and have the same problem. My script looks 32 times on "[salt-master not working yet]" then gives up.
We're cutting one today.
I'm still seeing the same issue as before with the new release, my master starts then times out waiting for salt-master to start. Do you know some spots I can look at to help troubleshoot? Do you want a new issue created to track against?
Please file a new issue and tag thanks. I'm sorry it didn't fix things for you! On May 22, 2015 4:49 AM, "Steve Sloka" wrote:


Question: local -a nonstatics=() local -a tests=() <ins> V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" </ins> for binary in "${binaries[@]}"; do if [[ "${binary}" =~ ".test"$ ]]; then

Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs.

Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?


Question: }, { <del> name: "spec:v6 ip:v6 dualstack:on", </del> <ins> name: "spec:v6 ip:v6 dualstack:on isIPv6Primary:false", isIPv6Primary:          false, </ins> specExpctPrimary:       false, clusterIPExpectPrimary: false, enableDualStack:        true,

What happened: When K8s is started in dual-stack mode with IPv6 as the "main" (or base) family services specified with ipFamily:IPv4 still get a ipv6 ; The is expicitly set to "10.0.0.0" using . What you expected to happen: Services with ipFamily:IPv4 shall get ipv4 ClusterIP addresses. How to reproduce it (as minimally and precisely as possible): Setup a K8s cluster in dual-stack mode with family IPv6 as base Define a service with ipFamily: IPv4 To make K8s think IPv6 is the main family the cue is to define ipv6 addresses to K8s binaries. Below are my settings as an example. I do not use but install "from scratch"; Master processes; is accessed with ipv4 which does not affect the K8s family. Worker processes; Anything else we need to know?: There are other problems with dual-stack/ipv6-base but may be caused by this one. For example does not work and some processes does not become "ready". Ipv6 traffic to ipv6 services and DNS works; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: - OS (e.g: ): BusyBox based own setup. Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): Others:
/sig network /cc
The problem seem to be; The problem occurs when an ipv6 range is specified first, even in an ipv4-based dual-stack cluster. But the ipv6 range must be specified first otherwise the "kubernetes" service gets an ipv4 address but an ipv6 backend; ~Perhaps the way to fix this problem is to make sure the service address is from the right family and require that the order to is "ipv4,ipv6".~ Bad idea!
There seem to be a option; But I can't see it in the help printout. I will try to use it...
:disappointed:
Here it where the kubernetes service address is set;
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: package kubelet import ( <ins> "fmt" </ins> "net" "reflect" "strings" "testing" "k8s.io/kubernetes/pkg/api/v1" <ins> "k8s.io/kubernetes/pkg/client/record" </ins> "k8s.io/kubernetes/pkg/util/bandwidth" )

Kubernetes version v1.3.0 Kubelet concatenates host and cluster domain entries for my pod, resulting in an invalid config (too many entries with duplicates). The max is only 6 search domains. Any more and DNS resolution breaks when trying to leverage search domains. The example below has only cluster DNS entries, but illustrates the issue: Host search nameserver 10.233.0.2 nameserver 10.90.0.1 options timeout:2 options attempts:2 Container kubectl run -i --tty busybox --image=busybox -- sh (skip waiting text) / # cat search nameserver 10.233.0.2 options attempts:2 options ndots:5 I should be enable cluster DNS config in my host DNS settings and still have functional pod DNS. Kubelet should be intelligent enough to filter out duplicate search domain entries (or drop search domains beyond the 6th entry)
I think some sane checks needs to be too. libc limits search field to 6 records and 255(?) characters. This limitation should be checked before creation.
Hey guys, I'm working on it.
Dup of


Question: token, err := GetServicePrincipalToken(config, env) assert.NoError(t, err) <del> oauthConfig, err := adal.NewOAuthConfig(env.ActiveDirectoryEndpoint, config.TenantID) </del> <ins> oauthConfig, err := adal.NewOAuthConfigWithAPIVersion(env.ActiveDirectoryEndpoint, config.TenantID, nil) </ins> assert.NoError(t, err) spt, err := adal.NewServicePrincipalToken(*oauthConfig, config.AADClientID, config.AADClientSecret, env.ServiceManagementEndpoint)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Due to the adal SDK implementation, the AAD token obtained by cloud provider from AzureAD token endpoint would have audience prefix. There is an extensive documentation in . The original contains fixes to kubectl as well as cloud provider. This issue will be targeted just to azure cloud provider. This wouldn't have the same impact as What you expected to happen: Change the SDK usage so that the AAD token's audience claim does not have prefix How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area provider/azure /sig cloud-provider
/cc


Question: podStatusProvider: mockPodStatusProvider{}, } <del> pod := makePod("1000", "1000") </del> <ins> pod := makePod("2", "2") </ins> container := &pod.Spec.Containers[0] err := mgr.AddContainer(pod, container, "fakeID") if !reflect.DeepEqual(err, testCase.expErr) { t.Errorf("CPU Manager AddContainer() error (%v). expected error: %v but got: %v", testCase.description, testCase.expErr, err) } <ins> if !testCase.expCPUSet.Equals(mgr.state.GetDefaultCPUSet()) { t.Errorf("CPU Manager AddContainer() error (%v). expected cpuset: %v but got: %v", testCase.description, testCase.expCPUSet, mgr.state.GetDefaultCPUSet()) } </ins> } }

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work?

I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!

Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs?

If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?


Question: } } <ins> func resourcePodSpecWithoutNodeName(memory, cpu string) v1.PodSpec { return v1.PodSpec{ Containers: []v1.Container{{ Resources: v1.ResourceRequirements{ Requests: allocatableResources(memory, cpu), }, }}, } } </ins> func allocatableResources(memory, cpu string) v1.ResourceList { return v1.ResourceList{ v1.ResourceMemory: resource.MustParse(memory),

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it.

The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign


Question: if [[ $(id -u) != 0 ]]; then sudo=sudo fi <del> "${sudo}" "${KUBE_ROOT}/third_party/multiarch/qemu-user-static/register/register.sh" --reset </del> <ins> ${sudo} "${KUBE_ROOT}/third_party/multiarch/qemu-user-static/register/register.sh" --reset </ins> curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/"${QEMUVERSION}"/x86_64_qemu-"${QEMUARCHS[$arch]}"-static.tar.gz | tar -xz -C "${temp_dir}" # Ensure we don't get surprised by umask settings chmod 0755 "${temp_dir}/qemu-${QEMUARCHS[$arch]}-static"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: login system root, and run below command from "test/images" dir, Will fail with below error message, ./image- line 91: : command not found This is cause by the empty string like "" cannot be understood as a shell command. What you expected to happen: image should be built successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ):all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: cd $ $ make ../image- bin webhook ../image- line 145: GOARM: unbound variable Makefile:23: recipe for target 'bin' failed make: [bin] Error 1 What you expected to happen: The binary or image should be built successfully. How to reproduce it (as minimally and precisely as possible): see my desc above, type "make" will see the error message. Anything else we need to know?: Environment*: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): Ubuntu Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I'm pretty sure this is supposed to be built via from which does default this environment variable.
, thanks for the information! I think the harness of the shell scripts could bring us a little more, GOARM should only been set when the ARCH is ARM Make a binary is possible where a Makefile is presented
That seems reasonable, but we should probably avoid computing this in many places, currently the GOARM=${GOARM:-7} is going to always be 7 when building in the currently supported way. We should change that if we're going to compute it elsewhere. cc-ing for test/images fun ...


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/uuid" utilfeature "k8s.io/apiserver/pkg/util/feature"

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: violatingVictims, nonViolatingVictims := filterPodsWithPDBViolation(potentialVictims.Items, pdbs) reprievePod := func(p *v1.Pod) bool { addPod(p) <del> fits, _, _ := podFitsOnNode(pod, meta, nodeInfoCopy, fitPredicates, nil, queue, false) </del> <ins> fits, _, _ := podFitsOnNode(pod, meta, nodeInfoCopy, fitPredicates, nil, queue, false, nil) </ins> if !fits { removePod(p) victims = append(victims, p)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: t.Errorf("got cache %v, want %v", got, cache) } } <ins> type MockTransport struct { res *http.Response } func (t *MockTransport) RoundTrip(req *http.Request) (*http.Response, error) { return t.res, nil } func TestClearingCredentials(t *testing.T) { fakeExpiry := time.Now().Add(time.Hour) cache := map[string]string{ "access-token": "fakeToken", "expiry":       fakeExpiry.String(), } cts := cachedTokenSource{ source:      nil, accessToken: cache["access-token"], expiry:      fakeExpiry, persister:   nil, cache:       nil, } tests := []struct { name  string res   http.Response cache map[string]string }{ { "Unauthorized", http.Response{StatusCode: 401}, make(map[string]string), }, { "Authorized", http.Response{StatusCode: 200}, cache, }, } persister := &fakePersister{} req := http.Request{Header: http.Header{}} for _, tc := range tests { authProvider := gcpAuthProvider{&cts, persister} fakeTransport := MockTransport{&tc.res} transport := (authProvider.WrapTransport(&fakeTransport)) persister.Persist(cache) transport.RoundTrip(&req) if got := persister.read(); !reflect.DeepEqual(got, tc.cache) { t.Errorf("got cache %v, want %v", got, tc.cache) } } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl "does not allow access" Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP GKE 1.4.5 OS (e.g. from /etc/os-release): Not sure, using gci Kernel (e.g. ): Not sure, using gci Install tools: GKE web UI Others: N/A What happened: In juggling multiple gcloud accounts with GKE clusters I accidentally used with the wrong application default credentials and it cached bad creds in After pointing gcloud at the right application default credentials and re-running , it continued trying and failing to use the bad creds cached in ; at this point I was stuck and confused for a few hours To repair, I had to manually edit to remove , and then started working again (or else wait ~1h for the access token to expire and refresh itself) What you expected to happen: After pointing gcloud at the right application default credentials should just work, or at least provide an error msg to the user that can help them figure out that they need to revoke the bad access token and how to do it How to reproduce it (as minimally and precisely as possible): Good scenario: just works Bad scenario: user makes one mistake and gets stuck
+1 Removing helped, thanks.
I can look into fixing this if no-one else has already made a start on it?
feel free to take it, thanks!

Ok, I think I have an understanding of what is happening call to is storing credentials in a json file in a known location. a subsequent credential-requiring kubectl command, the GCP plugin is being loaded, and the application default credentials are being exchanged for a token. That token is then cached in the .kube/config file. So where do things go wrong? As far as I can tell, if I switch to a cluster-2 without refreshing the application default credentials from it's separate account, those existing credentials will still be exchanged successfully for an oauth2 token and cached against cluster-2. Those tokens will then fail against a call to cluster-2 - because they were intended for cluster-1. The only sure-fire fix I can think of would be to ensure the application default credentials client ID is stored in users[].user.authprovider. Doing this would at least enable the ability to verify that the default credentials were valid for that particular cluster, and if not, prompt the user with a message to correctly aquire the application default credentials for it. Lets call this option 1. Of course, there is a problem here with determining what the Client ID is in the first place. I think this would require that a call to would store the client ID in the generated config. This would require exposing the client id from oauth2/google. Otherwise, perhaps storing the client ID the first time around would be enough - assuming the user at least manages to authenticate successfully against the cluster in the first instance. Then only oauth2/google would require a change, and not the gcloud command. The other option (option 2) would be, upon failure, check if is true and if-so, prompt the user that their application default credentials may be invalid for the target cluster, and to either prompt or clear the currently cached credentials in .kube/config. This would probably mean extending the auth provider interface such that a function may be provided that can be called upon failure that enables the auth provider plugin to do some checking of the state and to provide a reasonable message as to what might be wrong with the credentials that have been provided. I feel like option 2 is probably the better option in the long term. Different auth plugins are likely to have different failure modes. Providing a way to communicate back to the user about possible edge cases seems a little more flexible. Any thoughts?

edit: On closer inspection this seems possible by wrapping the oauth2.Transport call in WrapTransport. I would then be able to check for an authorization failure and provide a reasonable error message based on the existing configuration.
Just come across this, its particularly strange to have to edit the config file, when there is a folder in the folder.
Is it really fixed? I still can reproduce it on k8s 1.14 with description provided in initial comment.
I also reproduced this, unfortunately. Tried to use kubectl to access a cluster from an unauthorised account. Then, switched to the authorised account but the invalid authentication was cached. To resolve it, I set the to an expired date.
Believe I'm hitting this issue as well
I just ran into this as well, gcloud 311.0.0, kubectl 1.18.6.
I've got the same issue. I need to change the expiry time to 1 year back as pointed out.
Solution that usually works for me So it will use different credentials when switching contexts
You'll need  installed ().


Question: testInfo := getTestInfo(spec) if testInfo != nil { testInfos = append(testInfos, testInfo) <del> } if err := validateTestName(testInfo.CodeName); err != nil { log.Fatal(err) </del> <ins> if err := validateTestName(testInfo.CodeName); err != nil { log.Fatal(err) } </ins> } } }

script result to a nil pointer dereference exception e.g. show the below exception, no exception found when the script got called. a test binary (which are a little behind the master) the spec on the latest master branch, the spec to yaml: No response <details</details<details</detailsN/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig apps /area conformance


Question: setEntrypointAndCommand(container, &dockerOpts) <ins> glog.V(3).Infof("Container %v/%v/%v: setting entrypoint "%v" and command "%v"", pod.Namespace, pod.Name, container.Name, dockerOpts.Config.Entrypoint, dockerOpts.Config.Cmd) </ins> dockerContainer, err := r.Client.CreateContainer(dockerOpts) if err != nil { if ref != nil {

Forked from downward, this test has been failing since go/k8s-test/kubernetes-e2e-gce/4042, which lines up well with . Assigning to to fix, though a revert may be in order soon (we've had too much noise).
The issue is that I didn't change the networking e2e to set 'Args' instead of 'Command'.
Clients shouldn't be using the internal representation.
hmmmm ... is this really fixed? not working for me atm.
What behaviors are you seeing?
This is a recent run of our Jenkins server:
Re-closing (that was a success I pasted).
This might be the test failure you mentioned over hangout. The detail information can be found at: The issue is fixed this morning. Thanks!


Question: // IsUnexpectedObjectError determines if err is due to an unexpected object from the master. // It supports wrapped errors and returns false when the error is nil. func IsUnexpectedObjectError(err error) bool { <del> uoe := &UnexpectedObjectError{} return err != nil && errors.As(err, &uoe) </del> <ins> uoe, ok := err.(*UnexpectedObjectError) return err != nil && (ok || errors.As(err, &uoe)) </ins> } // SuggestsClientDelay returns true if this error suggests a client delay as well as the

I have a controller that creates objects. It uses to notice when create fails due to a deleting namespace. This check was working well until I moved it away from each call. In some cases, the error is wrapped using causing the function to always return false. I expected to work on both wrapped and unwrapped errors, like the functions do. This short program shows the issue: I am able to work around it by first calling . This type assertion should be replaced with a call to : /sig api-machinery v1.22.5, but I also see the affected code in master. <details</detailsn/a n/a k3s n/a n/a
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: return nil, fmt.Errorf("[BUG] ELB can't have empty list of Security Groups to be assigned, this is a Kubernetes bug, please report") } <del> { </del> <ins> if setupSg { </ins> ec2SourceRanges := []*ec2.IpRange{} for _, sourceRange := range sourceRanges.StringSlice() { ec2SourceRanges = append(ec2SourceRanges, &ec2.IpRange{CidrIp: aws.String(sourceRange)})

What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).

All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return ret } <ins> // In addition to Unstructured objects (Custom Resources), we also may sometimes need to // decode unversioned Options objects, so we delegate to parameterScheme for such types. parameterScheme := runtime.NewScheme() parameterScheme.AddUnversionedTypes(schema.GroupVersion{Group: crd.Spec.Group, Version: crd.Spec.Version}, &metav1.ListOptions{}, &metav1.ExportOptions{}, &metav1.GetOptions{}, &metav1.DeleteOptions{}, ) parameterScheme.AddGeneratedDeepCopyFuncs(metav1.GetGeneratedDeepCopyFuncs()...) parameterCodec := runtime.NewParameterCodec(parameterScheme) </ins> kind := schema.GroupVersionKind{Group: crd.Spec.Group, Version: crd.Spec.Version, Kind: crd.Spec.Names.Kind} <ins> typer := unstructuredObjectTyper{ delegate:          parameterScheme, unstructuredTyper: discovery.NewUnstructuredObjectTyper(nil), } creator := unstructuredCreator{} </ins> storage := customresource.NewREST( schema.GroupResource{Group: crd.Spec.Group, Resource: crd.Spec.Names.Plural}, schema.GroupVersionKind{Group: crd.Spec.Group, Version: crd.Spec.Version, Kind: crd.Spec.Names.ListKind}, UnstructuredCopier{}, customresource.NewStrategy( <del> discovery.NewUnstructuredObjectTyper(nil), </del> <ins> typer, </ins> crd.Spec.Scope == apiextensions.NamespaceScoped, kind, ), r.restOptionsGetter, ) <del> parameterScheme := runtime.NewScheme() parameterScheme.AddUnversionedTypes(schema.GroupVersion{Group: crd.Spec.Group, Version: crd.Spec.Version}, &metav1.ListOptions{}, &metav1.ExportOptions{}, &metav1.GetOptions{}, &metav1.DeleteOptions{}, ) parameterScheme.AddGeneratedDeepCopyFuncs(metav1.GetGeneratedDeepCopyFuncs()...) parameterCodec := runtime.NewParameterCodec(parameterScheme) </del> selfLinkPrefix := "" switch crd.Spec.Scope { case apiextensions.ClusterScoped:

BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to.

What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.


Question: ExpectedListenerAddress: "127.0.0.1", }, { <del> Hostname:                "[::1]", </del> <ins> Hostname:                "::1", </ins> Protocol:                "tcp6", ShouldRaiseError:        false, ExpectedListenerAddress: "::1", }, { <del> Hostname:         "[::1]", </del> <ins> Hostname:         "::1", </ins> Protocol:         "tcp4", ShouldRaiseError: true, },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When using IPv6 addresses, the kubelet passes the address to client-go, which does not correctly format the address for use in API requests. For example: Failed to list v1.Pod: Get https://2001:129::136:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dnode136&resourceVersi on=0: dial tcp: too many colons in address 2001:129::136:6443 What you expected to happen: The URI should have https://[2001:129::136]:6443, so that SplitHostPort() will correctly be able to separate the host and port components. How to reproduce it (as minimally and precisely as possible): Create cluster using IPv6 address (e.g. kubeadm init --apiserver-advertise-address=2001:129::136). When the kubelet starts up, it'll log this error. Anything else we need to know?: This is being handled as part of effort to support IPv6 in Kubernetes. Was found by a co-worker, I'll work on a commit to fix. Environment: Kubernetes version (use ): Using latest, via KubeAdm Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-81-generic Install tools: Using KubeAdm Others:
/area client-libraries /area kubelet /area ipv6
/assign
The point of failure is here: SplitHostPort will accept IPv4 and IPv6 addresses, but the latter must have square brackets around the IPv6 address.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery because client-go
Should we open this on the repository?

I see in the client-go repo read me that it states: Does that confirm that I should do a PR here?
One thing that I was wonder, was whether or not the hostname/address provided in the call to getListener, has been validated. If so, and it can only be a hostname, IPv4, or IPv6 address as a string, we could just check if there is a ":" in the name to determine IPv6. Otherwise, I could use net.ParseIP(), and net.To4() to determine if IPv6. I'll start working with the latter assumption, but please let me know, if the former could be used.
Also, I'm making the assumption that we'd want to make client-go be able to handle IPv6 addresses, rather than having each user of client-go (e.g. kubelet) from ensuring that the IP address is of the right form. IOW, have client-go add the needed [] around IPv6 addresses, rather than have kubelet do that. My thought is that there may be cases where we want just the IP address and not the modified IP. Does that sound OK?


Question: addRunFlags(cmd, o) cmdutil.AddApplyAnnotationFlags(cmd) cmdutil.AddPodRunningTimeoutFlag(cmd, defaultPodAttachTimeout) <ins> // Deprecate the cascade flag. If set, it has no practical effect since the created pod has no dependents. // TODO: Remove the cascade flag from the run command in kubectl 1.29 cmd.Flags().MarkDeprecated("cascade", "because it is not relevant for this command. It will be removed in version 1.29.") // Deprecate and hide unused flags. // These flags are being added to the run command by DeleteFlags to support pod deletion after attach, // but they are not used if set, so they effectively do nothing. // TODO: Remove these flags from the run command in kubectl 1.29 cmd.Flags().MarkDeprecated("filename", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("force", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("grace-period", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("kustomize", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("recursive", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("timeout", "because it is not used by this command. It will be removed in version 1.29.") cmd.Flags().MarkDeprecated("wait", "because it is not used by this command. It will be removed in version 1.29.") </ins> return cmd }

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: } func (m *effectiveVersion) BinaryVersion() *version.Version { <ins> if m.useDefaultBuildBinaryVersion.Load() { return defaultBuildBinaryVersion() } </ins> return m.binaryVersion.Load() }

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: } client := fake.NewSimpleClientset(csr) <del> informerFactory := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) </del> <ins> informerFactory := informers.NewSharedInformerFactory(fake.NewSimpleClientset(csr), controller.NoResyncPeriodFunc()) </ins> handler := func(csr *certificates.CertificateSigningRequest) error { csr.Status.Conditions = append(csr.Status.Conditions, certificates.CertificateSigningRequestCondition{

Seen in happy to send a fix but didn't know if this might have been address with your current refactors. Output:
This looks new. I'd happily review a fix. Thanks.
Dupe of


Question: return nil, errors.New("unable to open query through DLL call") } <del> ret = win_pdh.PdhValidatePath(counter) if ret != win_pdh.ERROR_SUCCESS { return nil, fmt.Errorf("unable to valid path to counter. Error code is %x", ret) } </del> ret = win_pdh.PdhAddEnglishCounter(queryHandle, counter, 0, &counterHandle) if ret != win_pdh.ERROR_SUCCESS { return nil, fmt.Errorf("unable to add process counter. Error code is %x", ret)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Im trying to setup node on window server 2019, but kubelet fails with error "F0217 17:36:06. 3200 ] failed to run Kubelet: unable to valid path to counter. Error code is " What you expected to happen: kubelet runs successfully How to reproduce it (as minimally and precisely as possible): install windows server 2019 with ru_RU locale run kubelet Anything else we need to know?: Problem appearances on any non English windows os. Its caused by hard-coded performance counter names in PdhValidatePath expects localized counter name. Environment: Kubernetes version (use ): 13.3 Cloud provider or hardware configuration: n/a OS (e.g. from /etc/os-release): windows server 2019 Kernel (e.g. ): n/a Install tools: n/a Others: n/a
/sig windows
Same problem with German Windows. We even tried replacing the english counters with the german equivalents - no success.
Hey, this is still an issue in version 1.15.0. Will this ever be fixed? Are there any workarounds available?
Hi to fix error you need to comment out lines 53-56 in and rebuild. I have tested it on 1.13 and after commenting out that lines everything works.
It's been over 6 months since this issue was opened. Is there a PR or anything for this yet? I'm hitting this error in Windows Server 2019 as well.
/assign
opened an issue the same day as my comment above. Note: I was using the English version.
I have reproduced this problem in the French environment, but the English environment is normal, has your windows installed other languages? I looked at the example of npd on Microsoft's official website and found that it may be a problem with kubernetes code. I am following up with the community on slack. The microsoft offical example is : They dont call this function
It seems like that removing does not fix this issue.

also returns the same error Error logs in 1.18.2:
very interesting, can you please open a new issue and investigate our options if you can?
I remember the last time I modified it and passed the test on the French and Russian Windows operating systems, can you describe your environment information in detail?


Question: value: 100m - name: MEMORY_REQUEST value: 200Mi <ins> - name: CPU_LIMIT value: 300m </ins> - name: MEMORY_LIMIT <del> value: 300Mi </del> <ins> value: 500Mi </ins>

Currently CPU consumption is not capped, which can be problematic in some setups
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant. If k8s back plane cluster components on the nodes are not resource constrained, user workload deployed to those nodes have zero guarantees and can make no assumptions regarding available resources while they are executing. Mick's ticket is definitely something that warrants fixing, since we've been affected by this precise limitation ourselves in production.
GitHub didn't allow me to assign the following users: frapposelli. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err := g.snapshot(); err != nil { return result, err } <ins> trace.Step("Snapshoting scheduler cache and node infos done") </ins> <del> trace.Step("Basic checks done") </del> startPredicateEvalTime := time.Now() filteredNodes, failedPredicateMap, filteredNodesStatuses, err := g.findNodesThatFit(state, pod) if err != nil { return result, err } <ins> trace.Step("Computing predicates done") </ins> // Run "postfilter" plugins. postfilterStatus := g.framework.RunPostFilterPlugins(state, pod, filteredNodes, filteredNodesStatuses)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We have enabled in genericscheduler. However, the steps being traced seem outdated. In particular, the "snapshot" is expensive and should be traced. Other steps can be traced as well wherever appropriate, e.g., PreFilter. Why is this needed: Improve granularity of tracing. /sig scheduling
/assign


Question: }, }, }, <ins> { name: "node port service with protocol sctp and externalTrafficPolicy local", services: []*v1.Service{ makeTestService("ns1", "svc1", func(svc *v1.Service) { svc.Spec.Type = "NodePort" svc.Spec.ClusterIP = "10.20.30.41" svc.Spec.Ports = []v1.ServicePort{{ Name:     "p80", Port:     int32(80), Protocol: v1.ProtocolSCTP, NodePort: int32(3001), }} svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal }), }, endpoints: []*discovery.EndpointSlice{ makeTestEndpointSlice("ns1", "svc1", 1, func(eps *discovery.EndpointSlice) { eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{"10.180.0.1"}, NodeName:  utilpointer.StringPtr(testHostname), }, { Addresses: []string{"10.180.1.1"}, NodeName:  utilpointer.StringPtr("otherHost"), }} eps.Ports = []discovery.EndpointPort{{ Name:     utilpointer.String("p80"), Port:     utilpointer.Int32(80), Protocol: &sctpProtocol, }} }), }, nodeIPs: []net.IP{ netutils.ParseIPSloppy("100.101.102.103"), }, nodePortAddresses: []string{}, expectedIPVS: &ipvstest.FakeIPVS{ Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{ { IP:       "10.20.30.41", Port:     80, Protocol: "SCTP", }: { Address:   netutils.ParseIPSloppy("10.20.30.41"), Protocol:  "SCTP", Port:      uint16(80), Scheduler: "rr", }, { IP:       "100.101.102.103", Port:     3001, Protocol: "SCTP", }: { Address:   netutils.ParseIPSloppy("100.101.102.103"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, }, Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{ { IP:       "10.20.30.41", Port:     80, Protocol: "SCTP", }: { { Address: netutils.ParseIPSloppy("10.180.0.1"), Port:    uint16(80), Weight:  1, }, { Address: netutils.ParseIPSloppy("10.180.1.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "100.101.102.103", Port:     3001, Protocol: "SCTP", }: { { Address: netutils.ParseIPSloppy("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, }, }, expectedIPSets: netlinktest.ExpectedIPSet{ kubeNodePortSetSCTP: { { IP:       "100.101.102.103", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, }, kubeNodePortLocalSetSCTP: { { IP:       "100.101.102.103", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, }, }, expectedIptablesChains: netlinktest.ExpectedIptablesChain{ string(KubeNodePortChain): {{ JumpChain: "RETURN", MatchSet: kubeNodePortLocalSetSCTP, }, { JumpChain: string(KubeMarkMasqChain), MatchSet: kubeNodePortSetSCTP, }, { JumpChain: "ACCEPT", MatchSet: kubeHealthCheckNodePortSet, }}, }, }, </ins> } for _, test := range tests {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: - name: 50-network-config.conf content: | [Service] <ins> Environment=ETCDCTL_PEERS=http://<master-private-ip>:4001 </ins> ExecStartPre=/bin/bash -c "until curl http://<master-private-ip>:4001/v2/machines; do sleep 2; done" ExecStartPre=/usr/bin/etcdctl set /coreos.com/network/config '{"Network":"10.244.0.0/16", "Backend": {"Type": "vxlan"}}' - name: docker.service

Tried to run through this guide: On VMware, there is no eth1, so the flannel configuration dies. Once you get past that, there is also an issue where etcdctl doesn't properly update the flannel network settings because the etcd server isn't set to the master.


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1beta1 import ( "io" "net/http" "strings" "testing" "k8s.io/api/events/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes/scheme" restfake "k8s.io/client-go/rest/fake" ) func TestCreateWithEventNamespace(t *testing.T) { event := &v1beta1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1beta1.Event wantErr bool }{ { name:  "create event", ns:    "default", event: event, }, { name:    "create event with different namespace", ns:      "other", event:   event, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.CreateWithEventNamespace(tt.event) if (err != nil) != tt.wantErr { t.Errorf("CreateWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } func TestPatchWithEventNamespace(t *testing.T) { event := &v1beta1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1beta1.Event data    []byte wantErr bool }{ { name:  "patch event", ns:    "default", event: event, data:  []byte{}, }, { name:    "patch event with different namespace", ns:      "other", event:   event, data:    []byte{}, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.PatchWithEventNamespace(tt.event, tt.data) if (err != nil) != tt.wantErr { t.Errorf("PatchWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } func TestUpdateWithEventNamespace(t *testing.T) { event := &v1beta1.Event{ ObjectMeta: metav1.ObjectMeta{ Name:      "test", Namespace: "default", }, } cli := &restfake.RESTClient{ Client: restfake.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       io.NopCloser(strings.NewReader("{}")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), } tests := []struct { name    string ns      string event   *v1beta1.Event wantErr bool }{ { name:  "patch event", ns:    "default", event: event, }, { name:    "patch event with different namespace", ns:      "other", event:   event, wantErr: true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { e := newEvents(New(cli), tt.ns) _, err := e.UpdateWithEventNamespace(tt.event) if (err != nil) != tt.wantErr { t.Errorf("UpdateWithEventNamespace() error = %v, wantErr %v", err, tt.wantErr) return } }) } } </ins>

Follow up from We should cover events expansion - both for core v1: events v1beta: with unit tests - can you please take care of it?
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: try, I, want, to, have, a. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I want to have a try Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // newNodeTree creates a NodeTree from nodes. func newNodeTree(nodes []*v1.Node) *NodeTree { nt := &NodeTree{ <del> tree:           make(map[string]*nodeArray), exhaustedZones: sets.NewString(), </del> <ins> tree: make(map[string]*nodeArray), </ins> } for _, n := range nodes { nt.AddNode(n)

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: } func (rt *cacheRoundTripper) WrappedRoundTripper() http.RoundTripper { return rt.rt.Transport } <ins> // A sumDiskCache is a cache backend for github.com/gregjones/httpcache. It is // similar to httpcache's diskcache package, but uses SHA256 sums to ensure // cache integrity at read time rather than fsyncing each cache entry to // increase the likelihood they will be persisted at write time. This avoids // significant performance degradation on MacOS. // // See https://github.com/kubernetes/kubernetes/issues/110753 for more. type sumDiskCache struct { disk *diskv.Diskv } // Get the requested key from the cache on disk. If Get encounters an error, or // the returned value is not a SHA256 sum followed by bytes with a matching // checksum it will return false to indicate a cache miss. func (c *sumDiskCache) Get(key string) ([]byte, bool) { b, err := c.disk.Read(sanitize(key)) if err != nil || len(b) < sha256.Size { return []byte{}, false } response := b[sha256.Size:] want := b[:sha256.Size] // The first 32 bytes of the file should be the SHA256 sum. got := sha256.Sum256(response) if !bytes.Equal(want, got[:]) { return []byte{}, false } return response, true } // Set writes the response to a file on disk. The filename will be the SHA256 // sum of the key. The file will contain a SHA256 sum of the response bytes, // followed by said response bytes. func (c *sumDiskCache) Set(key string, response []byte) { s := sha256.Sum256(response) _ = c.disk.Write(sanitize(key), append(s[:], response...)) // Nothing we can do with this error. } func (c *sumDiskCache) Delete(key string) { _ = c.disk.Erase(sanitize(key)) // Nothing we can do with this error. } // Sanitize an httpcache key such that it can be used as a diskv key, which must // be a valid filename. The httpcache key will either be the requested URL (if // the request method was GET) or "<method> <url>" for other methods, per the // httpcache.cacheKey function. func sanitize(key string) string { // These keys are not sensitive. We use sha256 to avoid a (potentially // malicious) collision causing the wrong cache data to be written or // accessed. return fmt.Sprintf("%x", sha256.Sum256([]byte(key))) } </ins>

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: "Time": recent(maxStatsAge), // We don't limit system container memory. "AvailableBytes":  BeNil(), <del> "UsageBytes":      bounded(100*framework.Kb, 10*framework.Gb), "WorkingSetBytes": bounded(100*framework.Kb, 10*framework.Gb), "RSSBytes":        bounded(100*framework.Kb, 1*framework.Gb), </del> <ins> "UsageBytes":      bounded(100*framework.Kb, memoryLimit), "WorkingSetBytes": bounded(100*framework.Kb, memoryLimit), "RSSBytes":        bounded(100*framework.Kb, memoryLimit), </ins> "PageFaults":      bounded(1000, 1E9), "MajorPageFaults": bounded(0, 100000), })

/kind bug What happened: e2e node summary test fails on machines with more than 10Gb of ram. What you expected to happen: memory limit bounds to look at node capacity rather than hardcode an upper bound. How to reproduce it (as minimally and precisely as possible): have more ram on a box and run e2e-node FOCUS=Summary Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node


Question: package validation import ( <ins> "fmt" </ins> "github.com/robfig/cron" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deployed an cronJob with an empty and it defaulted to which is an invalid option: Leaving the key out completely has the same result What you expected to happen: The empty key to default to something valid like or . Also document what this default is, my proposal would be to have as default since some tasks can't be reran easily. But it might be nice to force as default to enforce the stateless mindset even for crons :) How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I guess not but I am open for questions :) Environment: Kubernetes version (use ): 1.19.2
My guess would be /sig api-machinery If it is better practice to wait for the triage instead of guessing the sig, let me know :+1:
/sig api-machinery
/assign
/remove-sig api-machinery /sig apps
/assign
/triage accepted


Question: if err != nil { return fmt.Errorf("rbd: failed to unmap device %s:Error: %v", device, err) } <ins> // load ceph and image/pool info to remove fencing if err := util.loadRBD(&rbd, mntPath); err == nil { // remove rbd lock util.defencing(rbd) } </ins> glog.Infof("rbd: successfully unmap device %s", device) } return nil

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { <del> _, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) </del> <ins> output, lastMetricsFetchErr = g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) </ins> return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr)

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // If version is not supported, the assertion fails with an error. // This test should be done early during the storage operation flow to avoid // unnecessary calls later. <ins> // `ver` argument holds the expected supported version. </ins> func (c *csiDriverClient) AssertSupportedVersion(ctx grpctx.Context, ver *csipb.Version) error { if c.versionAsserted { if !c.versionSupported {

/kind bug What happened: When a CSI driver reports a version of the CSI spec which does not match that stored in Kubernetes, Kubernetes prints the following message: What you expected to happen: Message should be: How to reproduce it (as minimally and precisely as possible): Use a CSI driver that reports version 0.0.0. Then deploy an application pod with a PVC from this driver. Application pod description will show the actual message above. Environment: Kubernetes version (use ): 1.9.1
/sig storage
/assign lpabon
hi, I can help fix that.
I did not see your PR until after I submitted.