Question: SyncFrequency:      5 * time.Second, HTTPCheckFrequency: 5 * time.Second, } <del> go otherKubelet.RunKubelet("", "", servers[0], "localhost", 0) </del> <ins> go otherKubelet.RunKubelet("", "", servers[0], "localhost", "", 0) </ins> // Ok. we're good to go. glog.Infof("API Server started on %s", apiserver.URL)

When using or with a key/value that has a multi-line string as value, that value will appear as quoted string in the log message, i.e. the log message will still be a single line. It would be nicer to print the value in text mode so that it goes across multiple lines, with clear indicators for start and end of the value. Indention could be used to avoid mistaking some embedded line as the start of a new log message. In JSON mode, the value should be formatted normally. /sig instrumentation /wg structured-logging Always printing as a single line can can make the log output unreadable ().
When adding something like this, can be reverted and implemented differently.
My original thinking was that only objects that explicitly ask for multi-line output get formatted that way. But that means that we need to review log calls very carefully. An alternative is to automatically switch from quoting strings to multi-line output based on some heuristic, for example when there's more than one line break.
Here's a potential implementation:
If structured logging is needed in any other case I can contribute, please inform if needed.
/triage accepted /assign /priority important-soon /area logging /cc /cc
We have the same problem as in kube-scheduler also in kubelet, it's just that no-one has seen and/or complained about the multi-line strings. With the benchmark + log analysis tool from I found the following multi-line string parameters in a kubelet log: <details</details


Question: cloudConfig.TokenSource = google.ComputeTokenSource("") cloudConfig.UseMetadataServer = true <ins> featureMap := make(map[string]bool) cloudConfig.AlphaFeatureGate = &AlphaFeatureGate{featureMap} </ins> if configFile != nil { if configFile.Global.ApiEndpoint != "" { cloudConfig.ApiEndpoint = configFile.Global.ApiEndpoint

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I created a service (type LoadBalancer) on a 1.8.3 cluster. What you expected to happen: A load balancer to be deployed to GCP How to reproduce it (as minimally and precisely as possible): On a v1.8.3 cluster with GCP integration, deploy a service type LoadBalancer. The controller-manager was started with . However, was not specified. Example service: Anything else we need to know?: Stack trace captured in gist: Environment: Kubernetes version (use ): v1.8.3 Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release): Ubuntu16 Kernel (e.g. ): 4.10.0-38-generic Install tools: Kubernetes-the-hard-way Others:
/sig gcp


Question: return dc.updateDeploymentStatus(allRSs, newRS, deployment) } <ins> // Wait for all old replica set to scale down to zero. if err := dc.waitForInactiveReplicaSets(activeOldRSs); err != nil { return err } </ins> // If we need to create a new RS, create it now // TODO: Create a new RS without re-listing all RSs. if newRS == nil {

The scenario is a single replica with an attached EBS. The create seems to happen right after the delete not waiting for the pod to be terminated. This results in the two kubelets fighting over the EBS which generally results in AWS becoming unhappy meaning the EBS has to be force detached. From the docs I would have thought deployments would wait until the pod is actually killed. Is this a bug or am i misinterpreting the docs?
As a work around, what happens if you use a rolling update with MaxSurge=0 and MaxUnavailable=1?
Just assigned for triage, not to fix the issue. Can clear your assignment once this is triaged.
Deployments with size of 1 should be the same (both in that it's really a Recreate and it has the same problem even though it only matters for Recreate), larger deployments should be rolling.
It's because Deployment scales old RS down to 0 (=0, and RS controller will observe this change later and then kill its pods) and then scales up the new one. So it's possible there's a race here.
Opened that partially fixes this.
Why it's closed? It's not fixed: Recreate still doesn't wait for termination.
what version are you using?
1.6.2-00 from deb package. I'm changing the Deployment to use different configmap. K8s recreates pod, but the new one can't listen on ports that are probably used by the old one (using ). Both are appearing in at the same time (old one is 'Terminating').
This regressed in 1.6 but was fixed again in I don't think it was cherry-picked back to 1.6 though and the cherry-pick is not clean. are you interested in backporting it to the release-1.6 branch?
the cherry-pick isn't clean mostly because of comments. Don't have time to actually build and test the PR though.
I think this is still an issue:
I cannot reproduce it anymore, can you provide a reproducer?
Not easily at the moment -- we ended up changing our pods to handle multiple running simultaneously gracefully.
Hi All, I'm still seeing this on the following version: I am using recreate and can see my current pod go into a terminating state and my new pod come up.

It is my understanding that the new pod should not come up until the old pod is terminated completely.
Seeing this on 1.14.3 too - AWS hardware went bad and stuff was in terminating (but still running) and our stateful singleton was brought back up alongside
We see the same problem with 1.14.3 where the old pod is Terminating and the new pod is in ContainerCreating even if we set the Got problem with Volume is already used by pod(s)...
The easiest way I've found to reproduce this is by using bash interactively - it swallows SIGTERM giving a bigger window to observe the behavior. One thing to note, the behavior is different with an explicit delete and one caused by a changing annotation. When changing an annotation, the new pod won't be scheduled until the old one is completely gone, but an explicit delete will create the new pod right away. That said, I don't know how it will behave with other events, say a node going away or being drained. Here is a way to reproduce it. Delete the pod () and watch the new one spin up right away.
recreate deployments wait for the termination of an old version only during a deployment upgrade. The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use . can you please provide a reproducer? Thanks.
Same issue here with version v1.17.6: Deployment with "Recreate" strategy: If I try to delete a pod, the old pod is still Terminating, the new pod has already get initialized and Running:
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see the comment right above you "recreate deployments wait for the termination of an old version only during a deployment upgrade.

The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use statefulsets."


Question: pod, err := c.CoreV1().Pods(ns).Create(pod) framework.ExpectNoError(err, "Failed to create pod: %v", err) defer func() { <ins> body, err := c.CoreV1().Pods(ns).GetLogs(pod.Name, &v1.PodLogOptions{}).Do().Raw() if err != nil { framework.Logf("Error getting logs for pod %s: %v", pod.Name, err) } else { framework.Logf("Pod %s has the following logs: %s", pod.Name, body) } </ins> framework.DeletePodOrFail(c, ns, pod.Name) }() framework.ExpectNoError(framework.WaitForPodSuccessInNamespaceSlow(c, pod.Name, pod.Namespace))

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O job in our CI using kubernetes master tests Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with mount options Since when has it been failing: since this PR has been merged: Testgrid link: link to our job is here: https://openshift-gce- https://openshift-gce- https://openshift-gce- Reason for failure: we don't know the reason of the failure but it doesn't seem related to CRI-O. It's constantly reproducible also. Anything else we need to know: /cc <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig node /sig storage
/cc
Curiously, it does not seem to have flaked in core kubernetes CI
I'm still looking at it on our side
This test is newly in this commit. It tests that nfs external dynamic provisioner can provision a volume with specific mount options and the volume is properly mounted in the pod. The mount options specified in this test are "vers=4.1", "proto=tcp", and "nosuid". () The test seems to fail to create a pod with the volume and other tests that don't specify these mount options don't fail. So, I guess that this is caused because this environment doesn't support either of the option above. Is it possible to try testing it manually by changing these options?
yes, I have a box where I can change them and re-test, do you have suggestions on what to use to test?
(sorry fat fingers)
If it was an unsupported mount option, then the test should fail every time, not just flake?
yes, it's failing every time for us
Are there kubelet node logs we can look at?
yes, we have artifacts for each job at https://gcsweb-
As a starting point, I would recommend deleting only "proto=tcp", and check if it still fails, first. Then moving back "proto=tcp" and deleting only "nosuid", and check what happens. And continue testing with combinations of options.

Or, if we can find harmless mount options that can be specified for nfs, we will be able to replace the options with certainly harmless ones. Note that tests for AWS and GCE uses "debug" and "nouid32", which don't seem to be valid. for nfs
The container is failing with exit-code 1 with: I'll try to see which one is the bad one
do you think we could make the test easier to debug and figure out which mount option doesn't work?
yes, I'm trying to add something to that command to at least have some logs if that fails cause right now, if you do kubectl get logs on the failing pod you don't get anything
looks like removing makes the container happy and test as well. I'm not sure why it fails on rhel and fedora though, I'll dig into that, but yeah, we could use something different if possible
confirmed that if I remove nosuid the test works fine, I"m modifying the test code to output something useful in case it fails
Thank you for your quick investigation and improvement!
I'm going to open a PR for the test output to have something useful () and understand why it fails with
sorry, I am the one who wrote that hideous bash one-liner. TY for any patches
so, isn't shown: is that required for the test? can we have another option checked instead?
maybe we can use ? who is setting and where?
PTAL at
do you guys have any insight on this?
As for this e2e test, we will be able to use any options which are valid for nfs e2e test, because the purpose of the test is to check that the volume mount options work well for provisioning and the options are properly reflected to the mount options inside the pod. (Actually, I chose for nfs test just because I found it first.) As for issue for nfs dynamic provisioner with this environment, it might be still worth looking into, if this option can be specified by users. However, this could be out of the scope of this issue.
exactly, thanks, I'm going to investigate it not as part of this PR
Since this test failure is non-blocking for release, setting priority approprately. /priority important-longterm


Question: return nil, err } options := metav1.PatchOptions{ <del> Force:  &obj.ForceConflicts, DryRun: []string{metav1.DryRunAll}, </del> <ins> Force:        &obj.ForceConflicts, FieldManager: obj.FieldManager, DryRun:       []string{metav1.DryRunAll}, </ins> } return resource.NewHelper(obj.Info.Client, obj.Info.Mapping).Patch( obj.Info.Namespace,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: processStartTime.WithLabelValues().Set(start) return nil } <del> func getProcessStart() (float64, error) { pid := os.Getpid() p, err := procfs.NewProc(pid) if err != nil { return 0, err } if stat, err := p.Stat(); err == nil { return stat.StartTime() } return 0, err } </del>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Start kubelet on a Windows node and you will get this error: What you expected to happen: The process start time to be read appropriately for Windows How to reproduce it (as minimally and precisely as possible): Start kubelet on a Windows node Anything else we need to know?: The gauge falls back to but we get this error. This information is avaliable via the windows package so we should be able to get an accurate time. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: azure OS (e.g: ): Windows 2019 Kernel (e.g. ): Version 10.0..1637 Install tools: aks-engine (but would be any) Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
/triage accepted


Question: }, }, { <del> podsPerNode: 35, </del> cpuLimits: framework.ContainersCPUSummary{ <del> stats.SystemContainerKubelet: {0.50: 0.12, 0.95: 0.16}, stats.SystemContainerRuntime: {0.50: 0.05, 0.95: 0.07}, }, // We set the memory limits generously because the distribution // of the addon pods affect the memory usage on each node. memLimits: framework.ResourceUsagePerContainer{ stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 70 * 1024 * 1024}, stats.SystemContainerRuntime: &framework.ContainerResourceUsage{MemoryRSSInBytes: 200 * 1024 * 1024}, }, }, { cpuLimits: framework.ContainersCPUSummary{ stats.SystemContainerKubelet: {0.50: 0.20, 0.95: 0.29}, stats.SystemContainerRuntime: {0.50: 0.06, 0.95: 0.09}, </del> <ins> stats.SystemContainerKubelet: {0.50: 0.35, 0.95: 0.50}, stats.SystemContainerRuntime: {0.50: 0.10, 0.95: 0.50}, </ins> }, podsPerNode: 100, <del> // We set the memory limits generously because the distribution // of the addon pods affect the memory usage on each node. </del> memLimits: framework.ResourceUsagePerContainer{ <del> stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 80 * 1024 * 1024}, </del> <ins> stats.SystemContainerKubelet: &framework.ContainerResourceUsage{MemoryRSSInBytes: 100 * 1024 * 1024}, </ins> stats.SystemContainerRuntime: &framework.ContainerResourceUsage{MemoryRSSInBytes: 300 * 1024 * 1024}, }, },

https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite} Previous issues for this test:
Looks like a duplicate of . Assigning it to for further triaging.
failing in https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- https://k8s-
dup
sorry, not a dup
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Kubelet [Serial] [Slow] [] regular resource usage tracking resource tracking for 35 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: 203.0.113.1	podFoo.domainFoo	podFoo # Entries added by HostAliases. <del> 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz 456.78.90.123	park 456.78.90.123	doo 456.78.90.123	boo </del> <ins> 123.45.67.89	foo	bar	baz 456.78.90.123	park	doo	boo </ins> `, }, }

What would you like to be : Currently, specifying multiple aliases for a single host using results in a line for each alias in . Example: leads to an file containing something like Why is this needed: Having multiple entries for the same IP address in can break DNS resolution for some software. See for more details <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig node


Question: manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady <del> jobSynced := false manager.syncHandler = func(ctx context.Context, jobKey string) (bool, error) { jobSynced = true return true, nil } </del> stopCh := make(chan struct{}) defer close(stopCh) <del> go sharedInformers.Core().V1().Pods().Informer().Run(stopCh) </del> <ins> podInformer := sharedInformers.Core().V1().Pods().Informer() go podInformer.Run(stopCh) cache.WaitForCacheSync(stopCh, podInformer.HasSynced) </ins> go manager.Run(context.TODO(), 1) // Create job but don't add it to the store. cases := map[string]struct { <del> job           *batch.Job inCache       bool wantJobSynced bool </del> <ins> job     *batch.Job inCache bool </ins> }{ "job_does_not_exist": { job: newJob(2, 2, 6, batch.NonIndexedCompletion),

pull-kubernetes-unit TestWatchOrphanPods/orphan May 4th Waiting for Pod to get the finalizer removed: timed out waiting for the condition No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Likely a bug in


Question: <ins> #!/bin/bash # Copyright 2015 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script dumps debugging state of currently running Kubernetes cluster. KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. echo "kube-dump.sh: Getting version..." "${KUBE_ROOT}/cluster/kubectl.sh" version echo echo "kube-dump.sh: Getting resources..." "${KUBE_ROOT}/cluster/kubectl.sh" get nodes,pods,rc,services,events -o json echo source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/${KUBERNETES_PROVIDER}/util.sh" detect-project &> /dev/null echo "kube-dump.sh: Getting docker statuses on all nodes..." ALL_NODES=(${MINION_NAMES[*]} ${MASTER_NAME}) for NODE in ${ALL_NODES[*]}; do echo "kube-dump.sh: Node $NODE:" ssh-to-node "${NODE}" ' sudo docker ps -a sudo docker images ' done echo echo "kube-dump.sh: Getting boundpods from etcd..." ssh-to-node "${MASTER_NAME}" ' ETCD_SERVER=$(hostname -i):4001 for DIR in $(etcdctl -C $ETCD_SERVER ls /registry/nodes); do echo "kube-dump.sh: Dir $DIR:" etcdctl -C $ETCD_SERVER get $DIR/boundpods done ' </ins>

When a user report an issue on Kubernetes, we normally ask them to provide some outputs by running kubectl commands, such as kubectl get <resource, kubectl describe <resourceetc. We ask for daemons' log from various nodes often. Sometimes the user has to reproduce the issue for us because the config is changed. Meanwhile the developer might spend a lot of time on debugging the issues on a deprecated version or mismatched client with server. I propose to add a dump command to kubectl, which collects related outputs, logs, from various components or related nodes, then generate a tarball and attach that to the issue.
cc
This seems very useful. It could be the starting point for a more comprehensive diagnose-me that the user runs when they see a problem.
For a while now the kubelet logs have been collected by node level logging. For Elasticsearch the user needs to enable cluster level logging (on by default for Elasticsearch node level logging). For gcp node level logging the kubelet logs will have been ingested into the cloud logging console for the GCE project used to host the cluster.
cc/ They are going to help with this issue. Thanks!
I guess this issue is not closed yet (it was incorrectly closed by keyword in PR message).
Together with we propose to dump the following information: level: description of all kubernetes resources: nodes, replication controllers, services, pods: kubectl get nodes,podes,rc,services,events -o json, level: docker processes and images on each master/minion node: docker ps -a, docker images. level: version of etcd and boundpods stored in it: etcd --version, etcdctl get boundpods. (may consume a lot of space): the whole state (dump) of etcd. (may consume a lot of space): logs of the following components: kubelet, docker, etcd, kube-apiserver and kube-addons. The information will be dump by hack/kube- script. The initial version of the script (PR ) implements point 1, 2 and 3.

We suggest extending the kube- by adding of the following flags: --all: dump etcd state and logs: in addition to points 1, 2 and 3, also dump information proposed in pointes 4 and 5 (this may consume a lot of space); --issue=XXX: instead of writing output to stdout, make a tarball and attach it to #XXX issue on kubernetes github.
Proposed command is (turning cluster-info into a command group).
- Is there anything else needed to be done in this PR? (can it be closed?)
The original proposal wasn't fully implemented, but a new more specific issue can be filed to record additional data.
cc/


Question: c.forget(false) c.stop() } <ins> glog.V(2).Infof("cacheWatcher add function blocked processing for %v", time.Since(startTime)) </ins> } func (c *cacheWatcher) sendWatchCacheEvent(event watchCacheEvent) {

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


Question: syncFn func(key string) error queue workqueue.RateLimitingInterface <ins> // last generation this controller updated the condition per CRD name (to avoid two // different version of the apiextensions-apiservers in HA to fight for the right message) lastSeenGenerationLock sync.Mutex lastSeenGeneration     map[string]int64 </ins> } // NewConditionController constructs a non-structural schema condition controller.

/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


Question: } gorestfulContainer := restful.NewContainer() <del> gorestfulContainer.ServeMux = http.NewServeMux() </del> gorestfulContainer.Router(restful.CurlyRouter{}) // e.g. for proxy/{kind}/{name}/{*} gorestfulContainer.RecoverHandler(func(panicReason interface{}, httpWriter http.ResponseWriter) { logStackOnRecover(s, panicReason, httpWriter)

whether should init gorestful.container.ServeMux once none No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
Hi , as for my understanding, func 'NewAPIServerHandler' initialize gorestfulContainer.ServeMux each time due to gorestfulContainer.ServeMux = http.NewServeMux() when the function is called ,am i right ?
Looks like it is duplicated and we should be able to remove the call. Feel free to assign if anyone would like to send a PR for this.
I want to work on this but need to some guidance
Yes, and it should already have been initialized when calling
yes got it , so we can remove call gorestfulContainer.ServeMux = http.NewServeMux() ?
yes
/assign I will raised PR.
Hi I have raised PR could you check ?


Question: out := normalizeLocation(in) out = stripTimes(out) out = stripAddresses(out) <ins> out = normalizeInitFunctions(out) </ins> if expected.NormalizeOutput != nil { out = expected.NormalizeOutput(out) }

Internal job that runs k8s UT with master golang. https://prow.ppc64le- UT is failing when run with golang tip version. Around 24th of August No response This being a negative test scenario, looking at the test log, it seems the failure message from rest client isn't matching the expected in the string at The part of the text that doesn't match is: Expected: Actual: Steps to reproduce: master golang version clone kubernetes/test/e2e/framework/pod test -v -run TestFailureOutput /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-test /kind cleanup
is it happening in x86 too? is a golang bug ? there is also another bigger problem
Yes it is happening in x86 too. Yes, it could even be a golang bug too. But thought some upgrades in golang might require modification here in k8s. The above panic error that you pointed out already has an issue
This issue is introduced post addition of https://go- Global level functions are getting wrapped by synthetic init and so is getting changed to in actual traceback. Fix strategy has to be decided to support golang change mentioned above. Tagging others to request more inputs. /cc
^^
I think the output test is too strict: it shouldn't depend on the name given to anonymous functions. On the other hand, it should still ensure that functions like are printed as expected. This can be solved by further transforming the text during - I'll prepare a PR. /assign


Question: glog.Errorf("Specified Node IP not found in cloudprovider") return } <del> nodeAddresses = []v1.NodeAddress{*nodeIP} </del> } newNode := node.DeepCopy() newNode.Status.Addresses = nodeAddresses

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: case "udp": connectUDP(dest, timeout, udpData) case "sctp": <del> connectSCTP(dest, timeout) </del> <ins> connectSCTP(dest, timeout, sctpData) </ins> default: fmt.Fprint(os.Stderr, "Unsupported protocoln", protocol) os.Exit(1)

How to repro: an agnhost pod listening on sctp into the pod and use the connect command to probe against the same pod The stack trace that points to and this is from the vendored library , /kind bug /sig network /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Run an agnhost pod listening on sctpExec into the pod and use the connect command to probe against the same pod The stack trace that points to /sig network /help </details
/assign
/triage accepted
here we are immediately closing the connection after successful connect leading to EOF at server end (it is expecting data) causing the failure. We can tweak this to connect send some data just like UDP and then exit? Works locally writing "hostname" to the connection. [EDIT] For me it sometimes failed with EOF, sometime with nil pointer exception at . This reads sock opts via syscall and we immediately close the connection .. kind of a race between the client closing connection and server reading fds?
the panic seems a bug in that project, no? Also SCTP, has a establishment handshake, so the data should not be relevant in this case
returns fd and socket address, for is used to implement (always read from struct field). The SCTP implementation discards the returned socket address by call. , and is always a new syscall (). In our case the client immediately closes the connection, there can be chances where SCTPGETPEERADDRS will fail and will return nil in these cases. I think this is reasonable for their project. We call and panic is raised by calling on nil object. We can easily avoid this by not null check.

Also, I strongly believe that the client handling should be done in a separate go-routine instead of the main loop to avoid errors like this crashing the pod. [EDIT] The client in our case needs to send some data after connection, or else the server will throw EOF and we currently have assertions that use for errors.
panicing does not seem very reasonable It looks to me that the problem is that the SCTPConn struct needs to check if the connection is still valid before trying to access the file descriptor , see the std library implementation and then, all methods implement this check to me it is a bug on the implementation
The implementation is not causing any panic, it simply returns nil. Panic is caused when we try to call on nil object.
oh, sorry, my bad I should read your comment with more detail, is well explained there I recognize I have zero idea of how sctp works internally, maybe we can just be more resilient on our tests and code and detect if is nil and avoid getting into the complexity of this protocol ... sending some random data also sounds good, we do this for udp IIRC
are you able to send a PR to add those improvements to the test image?


Question: # cleanup kubectl delete svc prune-svc 2>&1 "${kube_flags[@]}" <ins> ## kubectl apply -f some.yml --force # Pre-condition: no service exists kube::test::get_object_assert services "{{range.items}}{{$id_field}}:{{end}}" '' # apply service a kubectl apply -f hack/testdata/service-revision1.yaml "${kube_flags[@]}" # check right service exists kube::test::get_object_assert 'services a' "{{${id_field}}}" 'a' # change immutable field and apply service a output_message=$(! kubectl apply -f hack/testdata/service-revision2.yaml 2>&1 "${kube_flags[@]}") kube::test::if_has_string "${output_message}" 'field is immutable' # apply --force to recreate resources for immutable fields kubectl apply -f hack/testdata/service-revision2.yaml --force "${kube_flags[@]}" # check immutable field exists kube::test::get_object_assert 'services a' "{{.spec.clusterIP}}" '10.0.0.12' # cleanup kubectl delete -f hack/testdata/service-revision2.yaml "${kube_flags[@]}" </ins> set +o nounset set +o errexit }

/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign


Question: }, expectedTopologyHints: nil, }, <ins> { description: "should not return preferred hints with multiple NUMA nodes for the pod with resources satisfied by a single NUMA node", assignments: state.ContainerMemoryAssignments{ "pod1": map[string][]state.Block{ "container1": { { NUMAAffinity: []int{0, 1}, Type:         v1.ResourceMemory, Size:         2 * gb, }, { NUMAAffinity: []int{0, 1}, Type:         hugepages2M, Size:         24 * mb, }, }, }, }, machineState: state.NUMANodeMap{ 0: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    1536 * mb, Free:           0, Reserved:       1536 * mb, SystemReserved: 512 * mb, TotalMemSize:   2 * gb, }, hugepages2M: { Allocatable:    20 * mb, Free:           0, Reserved:       20 * mb, SystemReserved: 0, TotalMemSize:   20 * mb, }, }, Cells:               []int{0, 1}, NumberOfAssignments: 2, }, 1: &state.NUMANodeState{ MemoryMap: map[v1.ResourceName]*state.MemoryTable{ v1.ResourceMemory: { Allocatable:    1536 * mb, Free:           gb, Reserved:       512 * mb, SystemReserved: 512 * mb, TotalMemSize:   2 * gb, }, hugepages2M: { Allocatable:    20 * mb, Free:           16 * mb, Reserved:       4 * mb, SystemReserved: 0, TotalMemSize:   20 * mb, }, }, Cells:               []int{0, 1}, NumberOfAssignments: 2, }, }, pod: getPod("pod2", "container2", &v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("1000Mi"), v1.ResourceMemory: resource.MustParse("1Gi"), hugepages2M:       resource.MustParse("16Mi"), }, Requests: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("1000Mi"), v1.ResourceMemory: resource.MustParse("1Gi"), hugepages2M:       resource.MustParse("16Mi"), }, }, ), systemReserved: systemReservedMemory{ 0: map[v1.ResourceName]uint64{ v1.ResourceMemory: 512 * mb, }, 1: map[v1.ResourceName]uint64{ v1.ResourceMemory: 512 * mb, }, }, expectedTopologyHints: map[string][]topologymanager.TopologyHint{ string(v1.ResourceMemory): { { NUMANodeAffinity: newNUMAAffinity(0, 1), Preferred:        false, }, }, hugepages2M: { { NUMANodeAffinity: newNUMAAffinity(0, 1), Preferred:        false, }, }, }, }, </ins> } for _, testCase := range testCases {

Setup: 3 Master nodes and 2 Worker Nodes. Version: v1.22.0-rc.0+ Memory Manager on Kubernetes Version (v1.22.0-rc.0+) and Nodes configured with Hugepages of size 2M on 2 Worker nodes. Worker node having 2 Numa Nodes (numa node0 and numa node1) of Hugepages of size 2M are 10 on each Numa node sh-4.4# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nrhugepages 10 sh-4.4# cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nrhugepages 10 After configuring Memory , where Hugepages are 20M (2M * 10) on each Numa node. we create QoS pod1 pod which consumes 24Mi Hugepages (of size 2M) Pod Spec: List pods Memory Manager state file: Now create a QoS Guaranteed Pod2 requesting 16Mi . On the same worker node. Pod spec: List pods: Memory Manager state file after pod2 is deployed. Pod2 should be rejected. Steps provided above Kubernetes version (use ): v1.22.0-rc.0+ Cloud provider or hardware configuration: Openshift OS (e.g: ):Red Hat Enterprise Linux CoreOS release 4.9 Kernel (e.g. ): Linux 4.18.0-305.12.1.rt7.84.el84.x8664 SMP PREEMPTRT Thu Jul 29 14:18:12 EDT 2021 x8664 x8664 x8664 GNU/Linux Kubelet configuration
/triage accepted /priority important-longterm /sig node /area kubelet
/assign


Question: } } } <del> metrics.Record(r, requestInfo, metrics.APIServerComponent, "", http.StatusTooManyRequests, 0, 0) </del> <ins> metrics.RecordRequestTermination(r, requestInfo, metrics.APIServerComponent, http.StatusTooManyRequests) </ins> tooManyRequests(r, w) } }

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: // parseField scans a field until a terminator func (p *Parser) parseField(cur *ListNode) error { p.consumeText() <del> var r rune for { r = p.next() if isTerminator(r) { p.backup() break } </del> <ins> for p.advance() { </ins> } value := p.consumeText() if value == "*" { cur.append(newWildcard()) } else { <del> cur.append(newField(value)) </del> <ins> cur.append(newField(strings.Replace(value, "", "", -1))) </ins> } return p.parseInsideAction(cur) } <ins> // advance scans until next non-escaped terminator func (p *Parser) advance() bool { r := p.next() if r == '' { p.next() } else if isTerminator(r) { p.backup() return false } return true } </ins> // isTerminator reports whether the input is at valid termination character to appear after an identifier. func isTerminator(r rune) bool { if isSpace(r) || isEndOfLine(r) {

I love using but I cannot figure out how to print an annotation that contains a special character like or or . This does not work: nor this: nor this: nor this: kubectl get pods tmp--dhpx0 -o jsonpath="{$['metadata']['annotations']['']}" (the last one should work according to what I can find about jsonPath, but doesn't, and anyhow is hard to get shell escaping right for. Need a way to quote a
A use case for this is extracting "created-by" or "init-container" annotations. I've wanted to do both in the last month and fought jsonpath.
FYI, this is possible with go-template. We should still fix the jsonpath though. Similar issue:


Question: kube::golang::build_some_binaries "${nonstatics[@]}" fi <ins> # Workaround for https://github.com/kubernetes/kubernetes/issues/115675 local testgogcflags="${gogcflags}" local testgoexperiment="${GOEXPERIMENT:-}" if [[ "$(go version)" == *"go1.20"* && "${platform}" == "linux/arm" ]]; then # work around https://github.com/golang/go/issues/58339 until fixed in go1.20.x testgogcflags="${testgogcflags} -d=inlstaticinit=0" # work around https://github.com/golang/go/issues/58425 testgoexperiment="nounified,${testgoexperiment}" kube::log::info "Building test binaries with GOEXPERIMENT=${testgoexperiment} GCFLAGS=${testgogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" fi </ins> for test in "${tests[@]:+${tests[@]}}"; do local outfile testpkg outfile=$(kube::golang::outfile_for_binary "${test}" "${platform}") testpkg=$(dirname "${test}") mkdir -p "$(dirname "${outfile}")" <del> go test -c  </del> <ins> GOEXPERIMENT="${testgoexperiment}" go test -c  </ins> ${goflags:+"${goflags[@]}"}  <del> -gcflags="${gogcflags}"  </del> <ins> -gcflags="${testgogcflags}"  </ins> -asmflags="${goasmflags}"  -ldflags="${goldflags}"  -tags="${gotags:-}"

master-blocking build-master Overall build is failing ! 2023-02-09 23:09:58 +0000 UTC make: [Makefile:487: cross] Error 1 !!! [0210 00:35:55] Call tree: !!! [0210 00:35:55] 1: kube::build::runbuildcommandex(...) !!! [0210 00:35:55] 2: kube::build::runbuildcommand(...) make: [Makefile:374: release] Error 1 level=fatal msg="failed to run: running make release: command /usr/bin/make release did not succeed: !!! [0210 00:35:54] Call tree:n!!! [0210 00:35:54] 1: hack/make- kube::golang::buildbinaries(...)nmake[1]: [Makefile:92: all] Error 1nmake: [Makefile:487: cross] Error 1n!!! [0210 00:35:55] Call tree:n!!! [0210 00:35:55] 1: kube::build::runbuildcommandex(...)n!!! [0210 00:35:55] 2: kube::build::runbuildcommand(...)nmake: * [Makefile:374: release] Error 1n" No response_ /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
looks like the same issue again: /cc
This would fall under sig-release or sig-testing? couldn't really gather much from logs about this though failure is so I guess sig-release then.
I'm going to reopen for that, which should work around the issue.
that's correct :) /remove-sig testing /sig release
The PR does not seem to fix the problem without disabling test cases. Opening a slack discussion and closing the loop:
Failing again.

/reopen
Reopened this issue. <detailsIn response to : Failing again. /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this is a different error: can we spawn a new issue to track rather than reopening the same one and mixing investigations for different issues?
Oh just the stack trace for make was the same, my bad, opened /close
Closing this issue. <detailsIn response to : Oh just the stack trace for make was the same, my bad, opened Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return err } o.AuthClient = client.AuthorizationV1() <ins> o.DiscoveryClient = client.Discovery() </ins> o.Namespace = "" if !o.AllNamespaces { o.Namespace, _, err = f.ToRawKubeConfigLoader().Namespace()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What you expected to happen: if returns true I expect to be able to perform the action without issue. Note in this case my service user has a rolebinding in a single namespace (and that is also what my kubecontext is set to.) if I change my default to a different namespace then the command returns false How to reproduce it (as minimally and precisely as possible): create a namespace (test) create a service account in that namespace create a role that gives get list and watch permissions to create a rolebinding from that role to the service account. Anything else we need to know?: this is special because namespaces are not scoped to a namespace. if you replace the command with then you get the exepcted result however this breaks the path of least suprise. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig auth
asks the authorization system if the given verb can be performed on the given noun at the given scope. It does not interpret whether the resource exists at that scope. We could warn if we find the specified resource and it does not exist at the specified scope (just like we warn if you ask about a resource that we can't find), but we would still interrogate the authorization system and return the results as we do today. /kind feature /remove-kind bug
/cc


Question: echo "${color_red}Could not find subnetwork with region ${REGION}, network ${NETWORK}, and project ${NETWORK_PROJECT}" } <ins> # Sets up Cloud NAT for the network. # Assumed vars: #   NETWORK_PROJECT #   REGION #   NETWORK function create-cloud-nat-router() { if [[ ${GCE_PRIVATE_CLUSTER:-} == "true" ]]; then gcloud compute routers create "$NETWORK-nat-router"  --project $NETWORK_PROJECT  --region $REGION  --network $NETWORK gcloud compute routers nats create "$NETWORK-nat-config"  --project $NETWORK_PROJECT  --router-region $REGION  --router "$NETWORK-nat-router"  --nat-all-subnet-ip-ranges  --auto-allocate-nat-external-ips fi } </ins> function delete-all-firewall-rules() { if fws=$(gcloud compute firewall-rules list --project "${NETWORK_PROJECT}" --filter="network=${NETWORK}" --format="value(name)"); then echo "Deleting firewall rules remaining in network ${NETWORK}: ${fws}"

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Attaches the disk to the kubelet's host machine. AttachDisk(b fcDiskMounter) (string, error) // Detaches the disk from the kubelet's host machine. <del> DetachDisk(disk fcDiskUnmounter, mntPath string) error </del> <ins> DetachDisk(disk fcDiskUnmounter, devName string) error </ins> } // utility to mount a disk based filesystem

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When user uses fc volume plugin, device file on operating system such as /dev/sdX remains after deleting a Pod. After deleting the Pod, PVC and PV, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up. The photon volume plugin already has this functionality in the removeFromScsiSubsystem(). Also the gce volume plugin has similar functionality in the udevadmChangeToDrive() too. photon: echo 1 /sys/block/<dev name/device/delete gce: udevadm trigger --action=change --property-match=DEVNAME=%s Also os-brick library in the OpenStack Cinder component uses same way with photon driver. What you expected to happen: The device file on operating system such as /dev/sdX will be cleaned up after deleting PVs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: KVM OS (e.g. from /etc/os-release): Fedora 25 Kernel (e.g. ):Install tools: local-up- Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
I'll work on this.
Reiterating the mentions to trigger a notification: <detailsIn response to : I'll work on this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // Skip non-memory backed volumes belonging to terminated pods volume := volumeToMount.VolumeSpec.Volume <ins> if volume == nil { continue } </ins> if (volume.EmptyDir == nil || volume.EmptyDir.Medium != v1.StorageMediumMemory) && volume.ConfigMap == nil && volume.Secret == nil { continue

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? panic, findAndRemoveDeletedPods Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): gci 56 (8938.0.0) Kernel (e.g. ): Install tools: Others: What happened: I created a pod and it went into the failed state. I couldn't find any info about the failure, so I logged into the GKE node and ran "journalctl -u kubelet" and saw: When I tail the log, the error messages continue indefinitely with multiple messages printed every second. Digging into I see: So I'm wondering if perhaps there is a bug where is in some cases. What you expected to happen: I did not expect to see the kublet log full of "invalid memory address or nil pointer dereference" errors. How to reproduce it (as minimally and precisely as possible): Sorry, I don't have a good way to reproduce this error. Anything else do we need to know: I'm running several pods, each with a GCE persistent disk mounted. When the pods finish, I run a different set of pods that mount the same persistent disks. I've seen various issues in past K8S versions where persistent disks don't get detached and reattached properly across K8S nodes, but so far with 1.5.1 it seems to have been working well up until now.
CC
guess it doesn't matter how it happens; it does. creating a fix.
If the pod is still around(even if failed state), can you paste output of . Just wondering how did the pod get in that state.
More information about the failed pod ():
looking at seems that there is a field for temporal volumes (configmaps, secrets, etc) and a field for persistent volumes and only one is set at a time.
Thanks for the quick PR!


Question: for _, minion := range list.Items { status, err := health.DoHTTPCheck(r.makeMinionURL(minion.ID), r.client) if err != nil { <del> glog.Errorf("%s failed health check with error: %s", minion, err) </del> <ins> glog.Errorf("%#v failed health check with error: %s", minion, err) </ins> continue } if status == health.Healthy {

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: # Find all of the built client binaries local platform platforms platforms=($(cd "${LOCAL_OUTPUT_BINPATH}" ; echo */*)) <del> for platform in "${platforms[@]}" ; do </del> <ins> for platform in "${platforms[@]}"; do </ins> local platform_tag=${platform///-} # Replace a "/" for a "-" <del> kube::log::status "Building tarball: client $platform_tag" </del> <ins> kube::log::status "Starting tarball: client $platform_tag" </ins> <del> local release_stage="${RELEASE_STAGE}/client/${platform_tag}/kubernetes" rm -rf "${release_stage}" mkdir -p "${release_stage}/client/bin" </del> <ins> ( local release_stage="${RELEASE_STAGE}/client/${platform_tag}/kubernetes" rm -rf "${release_stage}" mkdir -p "${release_stage}/client/bin" </ins> <del> local client_bins=("${KUBE_CLIENT_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then client_bins=("${KUBE_CLIENT_BINARIES_WIN[@]}") fi </del> <ins> local client_bins=("${KUBE_CLIENT_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then client_bins=("${KUBE_CLIENT_BINARIES_WIN[@]}") fi </ins> <del> # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # KUBE_CLIENT_BINARIES array. cp "${client_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/client/bin/" </del> <ins> # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # KUBE_CLIENT_BINARIES array. cp "${client_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/client/bin/" </ins> <del> kube::release::clean_cruft </del> <ins> kube::release::clean_cruft </ins> <del> local package_name="${RELEASE_DIR}/kubernetes-client-${platform_tag}.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." </del> <ins> local package_name="${RELEASE_DIR}/kubernetes-client-${platform_tag}.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." ) & </ins> done <ins> kube::log::status "Waiting on tarballs" wait || { kube::log::error "client tarball creation failed"; exit 1; } </ins> } # Package up all of the server binaries

Filing for This code is wrong The local needs to be on a separate line from the command execution, and we need to be resilient to not existing on OS X, like:
For once I was actually watching the output of a build and noticed that I was seeing an error about md5sum not being found. I'm building on a mac using boot2docker. The short_hash function () works fine on my mac as md5 does exist.
Actually, I'm just going to fix this as a follow-on to .
Thank you!


Question: // However, we need to tweak it e.g. to differentiate GET from LIST. verb := canonicalVerb(strings.ToUpper(req.Method), scope) if requestInfo.IsResourceRequest { <del> MonitorRequest(req, verb, requestInfo.APIGroup, requestInfo.APIVersion, requestInfo.Resource, requestInfo.Subresource, scope, component, contentType, code, responseSizeInBytes, elapsed) </del> <ins> requestTerminationsTotal.WithLabelValues(cleanVerb(verb, req), requestInfo.APIGroup, requestInfo.APIVersion, requestInfo.Resource, requestInfo.Subresource, scope, component, codeToString(code)).Inc() </ins> } else { <del> MonitorRequest(req, verb, "", "", "", requestInfo.Path, scope, component, contentType, code, responseSizeInBytes, elapsed) </del> <ins> requestTerminationsTotal.WithLabelValues(cleanVerb(verb, req), "", "", "", requestInfo.Path, scope, component, codeToString(code)).Inc() </ins> } }

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: // Run the tests klog.V(2).Infof("Starting tests on %q", host) podManifestPath := getPodPath(workspace) <del> cmd := fmt.Sprintf("'timeout -k 30s %fs docker run --rm --privileged=true --net=host -v /:/rootfs -v %s:%s -v %s:/var/result -e TEST_ARGS=--report-prefix=%s -e EXTRA_ENVS=%s %s'", timeout.Seconds(), podManifestPath, podManifestPath, results, junitFilePrefix, extraEnvs, getConformanceTestImageName(systemSpecName)) testOutput, err := SSH(host, "sh", "-c", cmd) if err != nil { return testOutput, err } return testOutput, nil </del> <ins> cmd := fmt.Sprintf("'timeout -k 30s %fs docker run --rm --privileged=true --net=host -v /:/rootfs -v %s:%s -v %s:/var/result -e TEST_ARGS=--report-prefix=%s -e EXTRA_ENVS=%s -e TEST_ARGS=--bearer-token=%s %s'", timeout.Seconds(), podManifestPath, podManifestPath, results, junitFilePrefix, extraEnvs, bearerToken, getConformanceTestImageName(systemSpecName)) return SSH(host, "sh", "-c", cmd) </ins> }

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: Type  string } <ins> var ( autoSubnetIPRange = &net.IPNet{ IP:   net.ParseIP("10.128.0.0"), Mask: net.CIDRMask(9, 32), } ) </ins> var providerIdRE = regexp.MustCompile(`^` + ProviderName + `://([^/]+)/([^/]+)/([^/]+)$`) func getProjectAndZone() (string, string, error) {

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: // ListKubeContainers lists running k8s CRI pods func (runtime *CRIRuntime) ListKubeContainers() ([]string, error) { <del> out, err := runtime.crictl("pods", "-q").CombinedOutput() </del> <ins> // Disable debug mode regardless how the crictl is configured so that the debug info won't be // iterpreted to the Pod ID. args := []string{"-D=false", "pods", "-q"} out, err := runtime.crictl(args...).CombinedOutput() </ins> if err != nil { return nil, errors.Wrapf(err, "output: %s, error", string(out)) }

prints out some warning messsage: And the pod on that node is not removed eventually. all the pods could be removed at last. Have a crictl installed on the node with the config like this, And then run below command, No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle /assign
The root cause is that when the debug mode is configured, will print out some debug message, e.g. DEBU[0000] get runtime connection DEBU[0000] ListPodSandboxRequest: &ListPodSandboxRequest{Filter:&PodSandboxFilter{Id:,State:nil,LabelSelector:map[string]string{},},} DEBU[0000] ListPodSandboxResponse: While our code interpret everything here as a Pod ID and try to rm them.
We cannot stop other to enable the debug mode for but we should ignore those debug message in our code.


Question: "fmt" v1 "k8s.io/api/core/v1" <del> schedulingv1 "k8s.io/api/scheduling/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" kubeapi "k8s.io/kubernetes/pkg/apis/core"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: const ( containerPollTime = 1 * time.Second <ins> lostPodPollTime   = 1 * time.Minute </ins> podRelistPeriod   = 5 * time.Minute )

In , there is a local constant named containerPollTime that is put to two very different uses --- waiting for a pod to come up, and recovering from pods getting "lost". The former needs to be very responsive, but the latter does not (but is still costly when the number of pods grows). The value is currently set to one second, and that makes both things happen frequently. It would be better if there were two different constants, so the monitoring for lost pods can be less frequent.


Question: return nil } return &priorityMetadata{ <del> podSelector:             getSelector(pod, pmf.serviceLister, pmf.controllerLister, pmf.replicaSetLister, pmf.statefulSetLister), podFirstServiceSelector: getFirstServiceSelector(pod, pmf.serviceLister), podTopologySpreadMap:    tpSpreadMap, </del> <ins> podSelector:          getSelector(pod, pmf.serviceLister, pmf.controllerLister, pmf.replicaSetLister, pmf.statefulSetLister), podTopologySpreadMap: tpSpreadMap, </ins> } } <del> // getFirstServiceSelector returns one selector of services the given pod. func getFirstServiceSelector(pod *v1.Pod, sl corelisters.ServiceLister) (firstServiceSelector labels.Selector) { if services, err := schedulerlisters.GetPodServices(sl, pod); err == nil && len(services) > 0 { return labels.SelectorFromSet(services[0].Spec.Selector) } return nil } </del> // getSelector returns a selector for the services, RCs, RSs, and SSs matching the given pod. func getSelector(pod *v1.Pod, sl corelisters.ServiceLister, cl corelisters.ReplicationControllerLister, rsl appslisters.ReplicaSetLister, ssl appslisters.StatefulSetLister) labels.Selector { labelSet := make(labels.Set)

/sig scheduling /priority important-soon See as an example. /assign
/unassign /help
/assign
/remove-help


Question: // HashContainer returns the hash of the container. It is used to compare // the running container with its desired spec. <ins> // Note: remember to update hashValues in container_hash_test.go as well. </ins> func HashContainer(container *v1.Container) uint64 { hash := fnv.New32a() <del> hashutil.DeepHashObject(hash, *container) </del> <ins> // Omit nil or empty field when calculating hash value // Please see https://github.com/kubernetes/kubernetes/issues/53644 containerJson, _ := json.Marshal(container) hashutil.DeepHashObject(hash, containerJson) </ins> return uint64(hash.Sum32()) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it.

There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?


Question: // Free the existing healthCheckNodePort and clear the HealthCheckNodePort field. case neededHealthCheckNodePort && !needsHealthCheckNodePort: glog.Infof("Transition to non LoadBalancer type service or LoadBalancer type service with ExternalTrafficPolicy=Global") <del> err := rs.serviceNodePorts.Release(int(oldHealthCheckNodePort)) if err != nil { glog.Warningf("error releasing service health check %s node port %d: %v", service.Name, oldHealthCheckNodePort, err) return false, errors.NewInternalError(fmt.Errorf("failed to free health check nodePort: %v", err)) } glog.Infof("Freed health check nodePort: %d", oldHealthCheckNodePort) </del> <ins> glog.V(4).Infof("Releasing healthCheckNodePort: %d", oldHealthCheckNodePort) nodePortOp.ReleaseDeferred(int(oldHealthCheckNodePort)) </ins> // Clear the HealthCheckNodePort field. service.Spec.HealthCheckNodePort = 0

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: healthcheck nodeport allocation doesn't have roll-back, this may lead to nodeport leak. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig network /assign Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network /assign
Could you clarity "roll-back"? It seems that we basically in the same way.
I didn't mean the HealthCheck NodePort release process in . I meant the allocation process in and . Take for example, please see: There is a defer for common NodePorts in : Once fails, the above defer will finally exec. Then the allocated NodePorts will roll-back, i.e. these NodePorts will be released. But the HealthCheck NodePort allocation process doesn't use this roll-back. This will lead to leak. I run a simple test and see the HealthCheck NodePort leak. Will paste the result later.
Testing service manifest, . some simple testing code. See: . Note that I add this error intentionally: kube-apiserver with flag . . We will get below output. This is expected. again. We will get below output. Note the error . That means that when creating service fails at step 3, the NodePort and HealthCheck NodePort should be both released, but the was NOT.
Fortunately, several minute later, the leak was repaired and cleaned up. But anyway, we should release the HealthCheck NodePort when fails. We shouldn't depend on the later repair.
Thanks for tracing down, sent for fixing.
We're having this issue on service deletion. It looks like healthcheck nodeport is not deallocated immediately when deleting the service. This is causing issues for backup and restore of services with heathcheck node port.


Question: `, coreDNSVersion: "1.3.1", }, <ins> { name: "Modified Corefile with no migration required", initialCorefileData: `consul { errors forward . 10.10.96.16:8600 10.10.96.17:8600 10.10.96.18:8600 { max_concurrent 1000 } loadbalance cache 5 reload } domain.int { errors forward . 10.10.0.140 10.10.0.240 10.10.51.40 { max_concurrent 1000 } loadbalance cache 3600 reload } .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp max_concurrent 1000 } cache 30 loop reload loadbalance } `, expectedCorefileData: `consul { errors forward . 10.10.96.16:8600 10.10.96.17:8600 10.10.96.18:8600 { max_concurrent 1000 } loadbalance cache 5 reload } domain.int { errors forward . 10.10.0.140 10.10.0.240 10.10.51.40 { max_concurrent 1000 } loadbalance cache 3600 reload } .:53 { errors health { lameduck 5s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf { prefer_udp max_concurrent 1000 } cache 30 loop reload loadbalance } `, coreDNSVersion: "1.6.7", }, </ins> } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { client := createClientAndCoreDNSManifest(t, tc.initialCorefileData, tc.coreDNSVersion) <del> // Get the Corefile and installed CoreDNS version. cm, corefile, currentInstalledCoreDNSVersion, err := GetCoreDNSInfo(client) </del> <ins> configMapBytes, err := kubeadmutil.ParseTemplate(CoreDNSConfigMap, struct{ DNSDomain, UpstreamNameserver, StubDomain string }{ DNSDomain:          "cluster.local", UpstreamNameserver: "/etc/resolv.conf", StubDomain:         "", }) </ins> if err != nil { <del> t.Fatalf("unable to fetch CoreDNS current installed version and ConfigMap.") </del> <ins> t.Errorf("unexpected ParseTemplate failure: %+v", err) </ins> } <del> err = migrateCoreDNSCorefile(client, cm, corefile, currentInstalledCoreDNSVersion) </del> <ins> err = createCoreDNSAddon(nil, nil, configMapBytes, client) </ins> if err != nil { <del> t.Fatalf("error creating the CoreDNS ConfigMap: %v", err) </del> <ins> t.Fatalf("error creating the CoreDNS Addon: %v", err) </ins> } migratedConfigMap, _ := client.CoreV1().ConfigMaps(metav1.NamespaceSystem).Get(context.TODO(), kubeadmconstants.CoreDNSConfigMap, metav1.GetOptions{}) if !strings.EqualFold(migratedConfigMap.Data["Corefile"], tc.expectedCorefileData) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: does not retaining coredns configmap using v1.19.4 makes no difference What you expected to happen: should follow documentation and retain coredns configmap How to reproduce it (as minimally and precisely as possible): upgrading from v1.18.6 to v1.19.3 Anything else we need to know?: works like a charm Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): CentOS 7 Kernel (e.g. ): 4.20.13- Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
seems to be a regression ?
/assign
Would you check you dns configmap for ? As , federation is removed from coredns. If you have federation enabled in your coredns configmap, the check may fail in
We don't have any parameter in our coredns configmap, or any kube-dns parameters in kubeadm config
Looking closely at the dry run messages above, this may not be a regression. The apparent reason it works in 1.18.12 is that support for CoreDNS 1.7.0 was not until 1.19. i.e. 1.18 sees an unsupported version of CoreDNS and therefore skips the corefile migration. Whereas in 1.19, CoreDNS 1.7.0 is supported, and the migration takes place. What does your complete Corefile look like before the migration?
If it's possible, can you also attach the entire dry run logs?
Sure.
Thanks.

I think the issue may be in the detection of the "default" configuration. That is, in the upgrade process, if it detects that you are using the "default" coredns corefile, it simply installs the new default over top of it, rather than migrating the existing one. I think since the default config stanza is in there in its original form, the whole Corefile is incorrectly being detected as a default. If i'm right, the fix could either go in the package (should at least eventually be fixed there), and/or kubeadm could just always migrate the existing config instead of deploying the default over top of the existing one.
I just tested against the package manually - and it correctly identifies the config as "non-default", so that's not it... And I also tested a migration on the Corefile, and the result included the extra stanzas as expected. So, the issue doesn't appear to lie in the migration package.
After testing with the code, I find the issue in , I fixed it in . A workaround for you is to create an empty kube-dns configmap. ———————————————————— edited new comments here I find still fix a problem , when kube-dns configmap is using federations.
Thanks, I'll try it
empty kube-dns configmap doesn't helped.
sorry for my mistake. According to my test, the create or update will trigger is no migrating. Even your corefile will meet some dep that need to be migrated. max_concurrent is new default. However, there is a typo in . Below is a workaround(not good one), add upstream to your corefile in kubernetes. ! It works for me.
if your current coredns is 1.6.7, upstream is deprecated but not removed. In new 1.7.0 coredns, upstream is removed.
thanks for the discussion and PRs this one is mergable for 1.21 as non critical: what do you think about this one and should we backport it as critical (also, to which k8s versions)?
/area kubeadm
It's not obvious to me how this relates to . The kubeadm dry run log appears to be replacing the existing config with the default. It should only be doing this if the existing config is semantically the same as a default Corefile, i.e.

if is true.
just explains that the migration is not triggered at all. if is not a fix for this issue, please remove this line from the PR description.
I think issue here is in/around , and how the result is interpreted... The code correctly assumes that if a Corefile is a default Corefile, then no migration is required. However later it incorrectly assumes the reciprocal: If migration is not required, then the corefile must be a default. Hence, it ends up replacing any Corefile that does not require migration with a default… even if it wasn't a default to begin with.
yes. The need be fixed as well, I will learn to fix it later if no one is assigned. I changed releaseNote to None.


Question: # Set up the context directory for the kube-build image and build it. function kube::build::build_image() { <ins> kube::build::ensure_tar </ins> local -r build_context_dir="${LOCAL_OUTPUT_IMAGE_STAGING}/${KUBE_BUILD_IMAGE}" kube::build::build_image_cross mkdir -p "${build_context_dir}" <del> tar czf "${build_context_dir}/kube-source.tar.gz" $(kube::build::source_targets) </del> <ins> "${TAR}" czf "${build_context_dir}/kube-source.tar.gz" $(kube::build::source_targets) </ins> kube::version::get_version_vars kube::version::save_version_vars "${build_context_dir}/kube-version-defs"

Since I've been having troubles with installation, I've digged a bit and found out that the permissions inside Kubernetes tarballs changed between 0.16.0 and 0.17.0. I've downloaded both tarballs, extracted them into separate directories and then extracted the Kubernetes server tarballs. This is what I get: I suppose, there has been a concious decision to restrict the world's access to those binaries. However, this breaks the installation routine mentioned above and probably makes life a bit more difficult in several other places. I'd be thankful for a correction if this wasn't intended or a pointer to the appropriate commit it this is legit -- I'd correct the installation.
assigning this to you to decide who the eventual assignee should be :)
Any news on this front? 0.20.1 is still restricted.
Apologies for the issue. It wasn't intentional -- likely an artifact of build scripts being run by different people? I'll check out 0.20.2 as I build it to make sure it has the right perms.
Apologies, but 0.20.2 didn't include fixed permissions. I now understand the issue, : "make release" builds a tarball using the permissions on files in your local git tree. I had "umask 027" set when I cloned, so those in-the-repo files had no world-readable perms. Setting "umask 022" before running "make release" is not enough -- it has to be part of creating the local tree when cloning. I'll update instructions and the next release (20.3 or 21.0) should fix this issue. And longer term, we need to put a bit more effort into releasing from a known-good environment and not rely on a developer's workstation's idiosyncrasies -- sorry this bit you.
I can't repro Trying again a different way, but I don't think the umask before clone should do anything (in particular, umasks aren't "sticky" on files, they're sticky to processes.)
And let's pretend on my personal fork isn't woefully out of date (who uses master on their personal fork? I should really just delete that that thing.) Comment updated.
Ah, I see the problem now.

works fine for the server tar, but the full tar also packages examples, etc., and for that, tar tends to just pick up the permissions of the git tree. Got it.
I'll take this. I'm going to write a script to build official bits, and this is easy enough to fix along the way.
Seems not fixed as of Kubernetes 1.0.7
sorry about that. is doing the 1.1.1 release and has been improving release scripts -- is 1.1.1 good re: permissions?
was not built by .
Reassigning to as release-oncall. Quinton, I can mentor or advise as necessary.
And yeah, is fine: There's no real bug here. There was a process issue that occurred with 1.0.7. We can rebuild the binaries if we need to.
FYI, leaving this issue closed, as per 's comment above. If anyone disagrees, please re-open this issue.


Question: Kind:       "Flunder", APIVersion: "wardle.k8s.io/v1alpha1", }, <del> ObjectMeta: metav1.ObjectMeta{Name: "test-flunder"}, </del> <ins> ObjectMeta: metav1.ObjectMeta{Name: flunderName}, </ins> Spec:       samplev1alpha1.FlunderSpec{}, } jsonFlunder, err := json.Marshal(testFlunder)

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: } uuidString := string(uuid.NewUUID()) <del> // UUID string is just hex separated by dashes, which is safe to // throw into regex like this pattern := "^" + uuidString + "$" </del> sentinelName := "__ratcheting_sentinel_field__" sch.Properties[sentinelName] = apiextensionsv1.JSONSchemaProps{ <del> Type:    "string", Pattern: pattern, // Put MaxLength condition inside AllOf since the string_validator // in kube-openapi short circuits upon seeing MaxLength, and we // want both pattern and MaxLength errors AllOf: []apiextensionsv1.JSONSchemaProps{ { MinLength: ptr((int64(1))), // 1 MinLength to prevent empty value from ever being admitted MaxLength: ptr((int64(0))), // 0 MaxLength to prevent non-empty value from ever being admitted }, }, </del> <ins> Type: "string", Enum: []apiextensionsv1.JSON{{ Raw: []byte(`"` + uuidString + `"`), }}, </ins> } for _, v := range myCRD.Spec.Versions {

integration TestRatchetingFunctionality/Enum TestRatchetingFunctionality/MinPropertiesMaxProperties TestRatchetingFunctionality/CELOptionalOldSelf TestRatchetingFunctionality/ TestRatchetingFunctionality/ since 3/8 No response /sig api-machinery
/assign /triage accepted


Question: printFlockerVolumeSource(volume.VolumeSource.Flocker, w) case volume.VolumeSource.Projected != nil: printProjectedVolumeSource(volume.VolumeSource.Projected, w) <ins> case volume.VolumeSource.CSI != nil: printCSIVolumeSource(volume.VolumeSource.CSI, w) </ins> default: w.Write(LEVEL_1, "<unknown>n") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl describe pod xxx not show csi inline volume, it shows <unknow! What you expected to happen: show csi inline volume normally like this ! How to reproduce it (as minimally and precisely as possible): a pod with inline csi volume describe pod xxx not show csi inline volume Anything else we need to know?: Environment: Kubernetes version (use ): 1.14 beta2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/assign
/sig cli