Question: done # Let the minion know who its master is <ins> # Recover the salt-minion if the salt-master network changes ## auth_timeout - how long we want to wait for a time out ## auth_tries - how many times we will retry before restarting salt-minion ## auth_safemode - if our cert is rejected, we will restart salt minion ## ping_interval - restart the minion if we cannot ping the master after 1 minute ## random_reauth_delay - wait 0-3 seconds when reauthenticating ## recon_default - how long to wait before reconnecting ## recon_max - how long you will wait upper bound ## state_aggregrate - try to do a single yum command to install all referenced packages where possible at once, should improve startup times ## </ins> mkdir -p /etc/salt/minion.d cat <<EOF >/etc/salt/minion.d/master.conf master: '$(echo "$MASTER_NAME" | sed -e "s/'/''/g")' <ins> auth_timeout: 10 auth_tries: 2 auth_safemode: True ping_interval: 1 random_reauth_delay: 3 state_aggregrate: - pkg </ins> EOF cat <<EOF >/etc/salt/minion.d/log-level-debug.conf

It appears that the provision-network script that is run on the kubernetes-master can cause salt provisioning on the minions to fail or block in a forever retry timeout error loop. I think it's because of the service network.restart that happens after openvswitch is installed in provision-network. I only say this because when I disable installing docker on kubernetes-master in vagrant, I do not have an issue. - is there a different way we can structure this on the kubernetes-master? For example, could we run the script in or something?


Question: func waitForPodRunningInNamespace(c *client.Client, podName string, namespace string) error { return waitForPodCondition(c, namespace, podName, "running", podStartTimeout, func(pod *api.Pod) (bool, error) { if pod.Status.Phase == api.PodRunning { <ins> Logf("Found pod '%s' on node '%s'", podName, pod.Spec.NodeName) </ins> return true, nil } if pod.Status.Phase == api.PodFailed {

Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of


Question: "testing" api "k8s.io/api/core/v1" <ins> "k8s.io/api/storage/v1beta1" </ins> meta "k8s.io/apimachinery/pkg/apis/meta/v1" utilfeature "k8s.io/apiserver/pkg/util/feature" fakeclient "k8s.io/client-go/kubernetes/fake"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: During I develop a driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. PS: I use cluster-driver register and make requiredAttach as false. kubectl get csidriver csi-xxx -o yaml ! What you expected to happen: Mount a raw block device successfully. How to reproduce it (as minimally and precisely as possible): driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. cluster-driver register and make requiredAttach as false. a pod with raw block devices Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: BaiduCloud OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /assign
I would send a PR later


Question: import ( "fmt" "regexp" <ins> "strings" </ins> azs "github.com/Azure/azure-sdk-for-go/storage" )

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: Ansible What happened: PVC keeps pending with a new created storage account What you expected to happen: Storage class can find storage account and then create a dynamic pv for pvc to bind How to reproduce it (as minimally and precisely as possible): a new storage account without create any container a storage class as below a pvc annotating this storage class pvc status is pending Anything else do we need to know: After I create a new container "vhds" in storage account of "qewestus", storage class works and pvc is bound
VHD container is a rerequiste. Will update the doc.
.. I was about to open this exact bug. Shouldn't the provisoner create the vhds container ?
yes, there is a todo for this case (also missing storage account)
Another minor issue is, new volumes take a lot of time (tens of mins) to be ext4 formatted. This causes the pod to timeout and fail upon first use! Any way to make this better
fix is in
Can anyone share how to create the vhds container? For those that don't yet have this fix
Easiest is through the portal, browse to your storage account and create a container called
Thats great thanks for the quick response. The automator in me now wants to see if I can automate the creation of that container through powershell ;)
Does the container have to be of type 'Container' ? IE Full public read access??

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Linux Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: hack/local-up- What happened: storage class pvc annotated with this storage class a pod consume this pvc read/write function with the mounted path pod/pvc the pv status which is "Failed" What you expected to happen: PV should not have "Failed" status before deleted How to reproduce it (as minimally and precisely as possible): Always Anything else do we need to know:
cc
Per message The disk is still being attached. As a result, PV cannot be deleted.
This condition is handled by the PV framework. See However the volume plugin must return which it looks like the Azure volume plugin is not doing.
please help remove label, kubectl can't do anything with this.


/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: <ins> #!bash # # bash completion file for core kubecfg commands # # This script provides completion of non replication controller options # # To enable the completions either: #  - place this file in /etc/bash_completion.d #  or #  - copy this file and add the line below to your .bashrc after #    bash completion features are loaded #     . kubecfg # # Note: # Currently, the completions will not work if the apiserver daemon is not # running on localhost on the standard port 8080 __contains_word () { local w word=$1; shift for w in "$@"; do [[ $w = "$word" ]] && return done return 1 } # This should be provided by the bash-completions, but give a really simple # stoopid version just in case.  It works most of the time. if ! declare -F _get_comp_words_by_ref >/dev/null 2>&1; then _get_comp_words_by_ref () { while [ $# -gt 0 ]; do case "$1" in cur) cur=${COMP_WORDS[COMP_CWORD]} ;; prev) prev=${COMP_WORDS[COMP_CWORD-1]} ;; words) words=("${COMP_WORDS[@]}") ;; cword) cword=$COMP_CWORD ;; -n) shift # we don't handle excludes ;; esac shift done } fi __has_service() { local i for ((i=0; i < cword; i++)); do local word=${words[i]} # strip everything after a / so things like pods/[id] match word=${word%%/*} if __contains_word "${word}" "${services[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then return 0 fi done return 1 } # call kubecfg list $1, # exclude blank lines # skip the header stuff kubecfg prints on the first 2 lines # append $1/ to the first column and use that in compgen __kubecfg_parse_list() { local kubecfg_output if kubecfg_output=$(kubecfg list "$1" 2>/dev/null); then out=($(echo "${kubecfg_output}" | awk -v prefix="$1" '/^$/ {next} NR > 2 {print prefix"/"$1}')) COMPREPLY=( $( compgen -W "${out[*]}" -- "$cur" ) ) fi } _kubecfg_specific_service_match() { case "$cur" in pods/*) __kubecfg_parse_list pods ;; minions/*) __kubecfg_parse_list minions ;; replicationControllers/*) __kubecfg_parse_list replicationControllers ;; services/*) __kubecfg_parse_list services ;; *) if __has_service; then return 0 fi compopt -o nospace COMPREPLY=( $( compgen -S / -W "${services[*]}" -- "$cur" ) ) ;; esac } _kubecfg_service_match() { if __has_service; then return 0 fi COMPREPLY=( $( compgen -W "${services[*]}" -- "$cur" ) ) } _kubecfg() { local opts=( -h -c ) local create_services=(pods replicationControllers services) local update_services=(replicationControllers) local all_services=(pods replicationControllers services minions) local services=("${all_services[@]}") local json_commands=(create update) local all_commands=(create update get list delete stop rm rollingupdate resize) local commands=("${all_commands[@]}") COMPREPLY=() local command local cur prev words cword _get_comp_words_by_ref -n : cur prev words cword if __contains_word "$prev" "${opts[@]}"; then case $prev in -c) _filedir '@(json|yml|yaml)' return 0 ;; -h) return 0 ;; esac fi if [[ "$cur" = -* ]]; then COMPREPLY=( $(compgen -W "${opts[*]}" -- "$cur") ) return 0 fi # if you passed -c, you are limited to create or update if __contains_word "-c" "${words[@]}"; then services=("${create_services[@]}" "${update_services[@]}") commands=("${json_commands[@]}") fi # figure out which command they are running, remembering that arguments to # options don't count as the command!  So a hostname named 'create' won't # trip things up local i for ((i=0; i < cword; i++)); do if __contains_word "${words[i]}" "${commands[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then command=${words[i]} break fi done # tell the list of possible commands if [[ -z ${command} ]]; then COMPREPLY=( $( compgen -W "${commands[*]}" -- "$cur" ) ) return 0 fi # remove services which you can't update given your command if [[ ${command} == "create" ]]; then services=("${create_services[@]}") elif [[ ${command} == "update" ]]; then services=("${update_services[@]}") fi case $command in create | list) _kubecfg_service_match ;; update | get | delete) _kubecfg_specific_service_match ;; *) ;; esac return 0 } complete -F _kubecfg kubecfg # ex: ts=4 sw=4 et filetype=sh </ins>

This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:


Question: utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol" clientv3 "go.etcd.io/etcd/client/v3" "k8s.io/klog/v2" )

TestChangeCRD " TestChangeCRD": 37, Recently, not sure the date. No response_ /sig api-machinery
also flaking in weirdly, this doesn't show up at all in
cc I can't tell if this is related to the openapi change or not
Actually, I think this is an issue either with the test or with watch behavior, not sure which. The test starts 10 watches, lets them run, then waits for them to exit cleanly, and instead, they are getting watch errors. I can't tell if the watch errors are expected or not, or if server or watch client behavior changed here or not. Opened to debug further
and stand out as a recent change to watching
cc on whether the watching changes in could explain this integration test getting watch errors it didn't previously get
also stands out
would explain this integration test getting watch errors it didn't previously get?
the filter is enabled only when the server run option has a positive time duration (the default is zero). I searched the k/k repo, only the unit tests in enables this filter: I did a quick scan of the logs, did not see any graceful termination activity either. I will take a deeper look at the logs
and all merged changes were limited to that filter?
This is a Go test, so the entire test output is treated as the failure message. That output is very different from one failure to the other, so in triage we probably have x different "clusters" with one failure in each, instead of one cluster with x failures. I don't know what triage does with such small clusters.
we have changes in only the the filter adds to the request context, I just searched the code base, did not find any other call site for by default, should block (on a nil channel) If the test is receiving an event, it implies that the following case is executing, right?
xref: ??
Thank you all for deflaking.
/triage accepted


Question: {ObjectMeta: metav1.ObjectMeta{Name: "nodeB", Labels: label2}}, {ObjectMeta: metav1.ObjectMeta{Name: "nodeC", Labels: label3}}, }, <ins> expectedAntiAffinity: topologyToMatchedTermCount{ {key: "region", value: "r1"}: 2, {key: "zone", value: "z11"}:  2, {key: "zone", value: "z21"}:  1, }, expectedAffinity: topologyToMatchedTermCount{}, </ins> }, { name: "preFilterState matching pod affinity and anti-affinity are updated correctly after adding and removing a pod", pendingPod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "pending", Labels: selector1}, <ins> Spec: v1.PodSpec{ Affinity: &v1.Affinity{ PodAffinity: affinityComplex, }, }, </ins> }, existingPods: []*v1.Pod{ {ObjectMeta: metav1.ObjectMeta{Name: "p1", Labels: selector1},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Maybe it should be What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Scheduling
I feel it is a valid error. But need confirmation from sig-scheduling maintainers. ... is it okay if I work on this?
ok
Yeah, this is a bug in preemption
/assign


Question: } func (a *poller) Run(stopCh <-chan struct{}) { <del> var failure int go wait.Until(func() { configuration, err := a.get() if err != nil { failure++ if failure >= a.failureThreshold { a.notReady() } return </del> <ins> go wait.Until(a.sync, a.interval, stopCh) } func (a *poller) sync() { configuration, err := a.get() if err != nil { a.failures++ if a.failures >= a.failureThreshold { a.notReady() </ins> } <del> failure = 0 a.setConfigurationAndReady(configuration) }, a.interval, stopCh) </del> <ins> return } a.failures = 0 a.setConfigurationAndReady(configuration) </ins> }

Found in I can reproduce locally using
/assign (via git blame)
cc
Thanks. I will take a look.
Got a fix locally. Testing it.


Question: } } <ins> func RecreatesPVCForPendingPod(t *testing.T, set *apps.StatefulSet, invariants invariantFunc) { client := fake.NewSimpleClientset() om, _, ssc := setupController(client) selector, err := metav1.LabelSelectorAsSelector(set.Spec.Selector) if err != nil { t.Error(err) } pods, err := om.podsLister.Pods(set.Namespace).List(selector) if err != nil { t.Error(err) } if _, err := ssc.UpdateStatefulSet(context.TODO(), set, pods); err != nil { t.Errorf("Error updating StatefulSet %s", err) } if err := invariants(set, om); err != nil { t.Error(err) } pods, err = om.podsLister.Pods(set.Namespace).List(selector) if err != nil { t.Error(err) } for _, claim := range getPersistentVolumeClaims(set, pods[0]) { om.claimsIndexer.Delete(&claim) } pods[0].Status.Phase = v1.PodPending om.podsIndexer.Update(pods[0]) if _, err := ssc.UpdateStatefulSet(context.TODO(), set, pods); err != nil { t.Errorf("Error updating StatefulSet %s", err) } // invariants check if there any missing PVCs for the Pods if err := invariants(set, om); err != nil { t.Error(err) } _, err = om.podsLister.Pods(set.Namespace).List(selector) if err != nil { t.Error(err) } } </ins> func TestStatefulSetControlScaleDownDeleteError(t *testing.T) { runTestOverPVCRetentionPolicies( t, "", func(t *testing.T, policy *apps.StatefulSetPersistentVolumeClaimRetentionPolicy) {

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//vendor/k8s.io/utils/mount:go_default_library", "//vendor/k8s.io/utils/net:go_default_library", "//vendor/k8s.io/utils/path:go_default_library", <ins> "//vendor/k8s.io/utils/strings:go_default_library", </ins> ] + select({ "@io_bazel_rules_go//go/platform:windows": [ "//pkg/kubelet/winstats:go_default_library",

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "fmt" "os" "path" <del> "path/filepath" </del> "syscall" "time"

Leads to this avc: In general, HostPath volumes in tests are bad news for selinux.


Question: if !strings.HasPrefix(mp, myMPSlash) { continue // skip -- not nested beneath myMountPoint } <del> if prevNestedMP != "" && strings.HasPrefix(mp, prevNestedMP) { </del> <ins> isNested := false for _, prevNestedMP := range prevNestedMPs { if strings.HasPrefix(mp, prevNestedMP) { isNested = true break } } if isNested { </ins> continue // skip -- double nested beneath myMountPoint } // since this mount point is nested, remember it so that we can check that following ones aren't nested beneath this one <del> prevNestedMP = mp + string(os.PathSeparator) </del> <ins> prevNestedMPs = append(prevNestedMPs, mp+string(os.PathSeparator)) </ins> retval = append(retval, mp[len(myMPSlash):]) } }

to have all the mount points sorted and grouped. For example, the following strings are sorted in this exact order: With the current implementation, test, we would get: This is worse on Windows, where paths usually have backslashes instead (with no test changes): [1] ` should group mount points correctly. /kind bug /sig windows Kubernetes master, Kubernetes v1.25 and older. N/A N/A <details</details<details</details<details</details
/triage accepted /milestone v1.26


Question: return nil, err } si := &serviceInfo{ <ins> isAliveAtomic:       1, </ins> proxyPort:           portNum, protocol:            protocol, socket:              sock,

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: http.Error(w, "", http.StatusNotFound) return } <del> json, err := runtime.Encode(r.codecs.LegacyCodec(), discoveryGroup) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if _, err := w.Write(json); err != nil { panic(err) } </del> <ins> responsewriters.WriteObjectNegotiated(r.codecs, schema.GroupVersion{}, w, req, http.StatusOK, discoveryGroup) </ins> }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): build--5-v1.7.5-3 Environment: Cloud provider or hardware configuration: Laptop (bare metal) OS (e.g. from /etc/os-release): Fedora 26 Kernel (e.g. ): Linux localhost.localdomain 4.11.0-0.rc6.git0.1.fc26.x8664 SMP Mon Apr 10 19:45:50 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: make && hack/local-cluster- -o _output/bin What happened: built from and ran with local-cluster- All commands gave the following error: Running showed the output: What you expected to happen: for commands to work, and Content-type to show How to reproduce it (as minimally and precisely as possible):
ccing
I see the text plain content type, but not kubectl errors
i was using the wrong version of kubectl. updated to new and i'm not getting an error any longer.
Text/plain coming back from the server is still an issue
cc
I'm not seeing against head and the pastebin is gone. Did you reproduce?
I did last week. Insecure and secured port behaved differently
You can reproduce this using against a Kubernetes cluster built from HEAD
Hi all, I think we could use write Content-type correctly. I have made a PR to fix this.


Question: #   KUBE_PASSWORD function get-password { # go template to extract the auth-path of the current-context user <del> local template='{{$ctx := index . "current-context"}}{{$user := index . "contexts" $ctx "user"}}{{index . "users" $user "auth-path"}}' </del> <ins> local template='{{with $ctx := index . "current-context"}}{{$user := index . "contexts" $ctx "user"}}{{index . "users" $user "auth-path"}}{{end}}' </ins> local file=$("${KUBE_ROOT}/cluster/kubectl.sh" config view -o template --template="${template}") <del> if [[ -r "$file" ]]; then </del> <ins> if [[ ! -z "$file" && -r "$file" ]]; then </ins> KUBE_USER=$(cat "$file" | python -c 'import json,sys;print json.load(sys.stdin)["User"]') KUBE_PASSWORD=$(cat "$file" | python -c 'import json,sys;print json.load(sys.stdin)["Password"]') return

contains function get-password. This uses a template which gets the current context. But a user who has not yet used the kubectl contexts feature has no context set. This causes a nil pointer deref while evaluating the template.
I think kube-up creates the needed entries in the .kubeconfig file. But it calls get-password before it does that. Need to allow the kubectl config view command to fail.
okay, I have a fix.


Question: // For reference: https://github.com/kubernetes/kubernetes/pull/81387 legacyregistry.RawMustRegister(grpcprom.DefaultClientMetrics) dbMetricsMonitors = make(map[string]struct{}) <ins> l, err := logutil.CreateDefaultZapLogger(etcdClientDebugLevel()) if err != nil { l = zap.NewNop() } etcd3ClientLogger = l.Named("etcd-client") } // etcdClientDebugLevel translates ETCD_CLIENT_DEBUG into zap log level. // NOTE(negz): This is a copy of a private etcd client function: // https://github.com/etcd-io/etcd/blob/v3.5.4/client/v3/logger.go#L47 func etcdClientDebugLevel() zapcore.Level { envLevel := os.Getenv("ETCD_CLIENT_DEBUG") if envLevel == "" || envLevel == "true" { return zapcore.InfoLevel } var l zapcore.Level if err := l.Set(envLevel); err == nil { log.Printf("Deprecated env ETCD_CLIENT_DEBUG value. Using default level: 'info'") return zapcore.InfoLevel } return l </ins> } func newETCD3HealthCheck(c storagebackend.Config, stopCh <-chan struct{}) (func() error, error) {

The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact.

What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).


Question: // 2. the ones passed in from framework.TestContext.ExtraEnvs // So this function needs to be called after the extra envs are applied. func updateImageWhiteList() { <del> // Union NodeImageWhiteList and CommonImageWhiteList into the framework image white list. framework.ImageWhiteList = NodeImageWhiteList.Union(commontest.CommonImageWhiteList) </del> <ins> // Union NodePrePullImageList and CommonImageWhiteList into the framework image pre-pull list. framework.ImageWhiteList = NodePrePullImageList.Union(commontest.CommonImageWhiteList) </ins> // Images from extra envs framework.ImageWhiteList.Insert(getNodeProblemDetectorImage()) framework.ImageWhiteList.Insert(getSRIOVDevicePluginImage())

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should come up with a better name for : Why is this needed: From /sig testing /priority important-soon /cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --We should come up with a better name for : From /priority important-soon /cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to help out with this. /assign
mind. It seems like you have a PR for it already. /unassign
And I misread the changes you made in . I'm back on this one. /assign
I'm going to rename this to . Please let me know if there are any objections or if there are things that I should be considering.
please feel free to take it. I'm only changing ImageWhileList withint
ultimately was changed to NodePrePullImageList because that's what it's actually used for, not for any sort of filtering.


Question: Skip("Nvidia GPUs do not exist on the node. Skipping test.") } <ins> framework.WaitForAllNodesSchedulable(f.ClientSet, framework.TestContext.NodeSchedulableTimeout) </ins> By("Creating the Google Device Plugin pod for NVIDIA GPU in GKE") devicePluginPod = f.PodClient().CreateSync(framework.NVIDIADevicePlugin(f.Namespace.Name))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The test has been failing since Nov 22nd. Looks like it is because the test restarts Kubelet and then calls getLocalNode() which checks for node readiness. Not sure why this problem only started to show up recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): COS Kernel (e.g. ): Install tools: Others:
/sig node
Hi, Just FYI, . Maybe it is helpful for you.


Question: seatsExpected: maximumSeats, }, { <del> name:       "request verb is list, resource version not set", requestURI: "http://server/apis/foo.bar/v1/events?limit=499", </del> <ins> name:       "request verb is list, has limit and resource version is 1", requestURI: "http://server/apis/foo.bar/v1/events?limit=399&resourceVersion=1", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, counts: map[string]int64{ <del> "events.foo.bar": 799, </del> <ins> "events.foo.bar": 699, </ins> }, <del> seatsExpected: 5, </del> <ins> seatsExpected: 8, </ins> }, { <del> name:       "request verb is list, continuation is set", requestURI: "http://server/apis/foo.bar/v1/events?continue=token&limit=499&resourceVersion=1", </del> <ins> name:       "request verb is list, limit not set", requestURI: "http://server/apis/foo.bar/v1/events?resourceVersion=1", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, counts: map[string]int64{ <del> "events.foo.bar": 799, </del> <ins> "events.foo.bar": 699, </ins> }, <del> seatsExpected: 5, </del> <ins> seatsExpected: 7, </ins> }, { <del> name:       "request verb is list, has limit", requestURI: "http://server/apis/foo.bar/v1/events?limit=499&resourceVersion=1", </del> <ins> name:       "request verb is list, resource version not set", requestURI: "http://server/apis/foo.bar/v1/events?limit=399", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, counts: map[string]int64{ <del> "events.foo.bar": 799, </del> <ins> "events.foo.bar": 699, </ins> }, <del> seatsExpected: 5, </del> <ins> seatsExpected: 8, </ins> }, { <del> name:       "request verb is list, resource version is zero", requestURI: "http://server/apis/foo.bar/v1/events?resourceVersion=0", </del> <ins> name:       "request verb is list, no query parameters, count known", requestURI: "http://server/apis/foo.bar/v1/events", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, counts: map[string]int64{ <del> "events.foo.bar": 799, </del> <ins> "events.foo.bar": 399, </ins> }, seatsExpected: 8, }, { <del> name:       "request verb is list, no query parameters, count known", </del> <ins> name:       "request verb is list, no query parameters, count not known", </ins> requestURI: "http://server/apis/foo.bar/v1/events", requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, <ins> countErr:      ObjectCountNotFoundErr, seatsExpected: maximumSeats, }, { name:       "request verb is list, continuation is set", requestURI: "http://server/apis/foo.bar/v1/events?continue=token&limit=399", requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, </ins> counts: map[string]int64{ <del> "events.foo.bar": 799, </del> <ins> "events.foo.bar": 699, </ins> }, seatsExpected: 8, }, { <del> name:       "request verb is list, no query parameters, count not known", requestURI: "http://server/apis/foo.bar/v1/events", </del> <ins> name:       "request verb is list, resource version is zero", requestURI: "http://server/apis/foo.bar/v1/events?limit=299&resourceVersion=0", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, <del> countErr:      ObjectCountNotFoundErr, seatsExpected: maximumSeats, </del> <ins> counts: map[string]int64{ "events.foo.bar": 399, }, seatsExpected: 4, </ins> }, { <del> name:       "request verb is list, resource version match is Exact", requestURI: "http://server/apis/foo.bar/v1/events?resourceVersion=foo&resourceVersionMatch=Exact&limit=499", </del> <ins> name:       "request verb is list, resource version is zero, no limit", requestURI: "http://server/apis/foo.bar/v1/events?resourceVersion=0", </ins> requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that does not make sense to me. Looking at how LIST requests are served (which is NOT easy, there is so much dependency injection), I concluded that the worst case number of objects touched in the apiserver equals the pagination limit only if all of the following are true. The LIST request is served from etcd rather than the watch cache. The pagination FeatureGate is enabled. The request has a specified limit and it is no greater than the number of objects in etcd (restricted by namespace if relevant). The request does not have a label selector or field selector. I expected the list work estimator to use the pagination limit only under the conditions listed above. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I commented on the PR.
Thank you both Mike and Wojciech! /triage accepted


Question: kl.statusManager.SetPodStatus(pod, apiPodStatus) <del> // Pods that are not runnable must be stopped - return a typed error to the pod worker if !runnable.Admit { klog.V(2).InfoS("Pod is not runnable and must have running containers stopped", "pod", klog.KObj(pod), "podUID", pod.UID, "message", runnable.Message) var syncErr error p := kubecontainer.ConvertPodStatusToRunningPod(kl.getRuntime().Type(), podStatus) if err := kl.killPod(ctx, pod, p, nil); err != nil { if !wait.Interrupted(err) { kl.recorder.Eventf(pod, v1.EventTypeWarning, events.FailedToKillPod, "error killing pod: %v", err) syncErr = fmt.Errorf("error killing pod: %w", err) utilruntime.HandleError(syncErr) } } else { // There was no error killing the pod, but the pod cannot be run. // Return an error to signal that the sync loop should back off. syncErr = fmt.Errorf("pod cannot be run: %v", runnable.Message) } return false, syncErr } </del> // If the network plugin is not ready, only start the pod if it uses the host network if err := kl.runtimeState.networkErrors(); err != nil && !kubecontainer.IsHostNetworkPod(pod) { kl.recorder.Eventf(pod, v1.EventTypeWarning, events.NetworkNotReady, "%s: %v", NetworkNotReadyErrorMsg, err)

Kubelet soft admission handlers were created as a way for the Kubelet to block pods that cannot be run for reasons that can be resolved. In practice, this mechanism was only ever used by AppArmor (). Problems with soft-admission rejection: is only used by AppArmor, and only for situations that cannot realistically be resolved (host or build does not support apparmor) held in the blocked state must be manually cleaned up. While there they hold reserved capacity from the scheduler and continually retry syncpod. other features work this way - usually Kubelet hard rejects pods that it knows it cannot run, or the runtime fails to start the pod/container. This was originally to prevent controllers from continuously recreating pods that cannot run, but that is a separate problem that should be approached globally, rather than with this 1-off AppArmor solution. I propose we remove Kubelet soft admission entirely, and move the AppArmor admission handler to the hard-admission list. /sig node /milestone v1.31 /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a different, alpha mechanism: . We could change AppArmor, seccomp and other mechanisms to work either with PodSchedulingContext or its eventual replacement. IMO we only need to sketch out an approach and see that it looks viable; we don't have to block graduation for AppArmor support on a fleshed-out design.
From my admittedly limited PoV I'm not against removing soft admission, and I got no opinions about AppArmor specifically. However this seems to connected to the recent conversation we're having in WG serving and SIG-node about retriable vs unretriable errors so I think we would need some coordination cc
+1
My understanding is that after this PR: soft admission just doesn't make sense any longer.
Hello ! Just a reminder, the code freeze is starting (about from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time.

As the issue is tagged for , is it still planned for this release?
Discussed in sig-node, decided to move forward with this (move AppArmor admission to the hard admission set, remove soft-admission code)
/assign
- do you want me to incorporate this change into


Question: } orphan := false switch { <ins> case options.OrphanDependents == nil && options.PropagationPolicy == nil && len(initialFinalizers) == 0: // if there are no deletion options, the default policy for replication controllers is orphan orphan = true </ins> case options.OrphanDependents != nil: // if the deletion options explicitly specify whether to orphan, that controls orphan = *options.OrphanDependents

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestStressingCascadingDeletion Testgrid link: Reason for failure: Maybe related: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig api-machinery
I'm seeing lots of these.
post-submit failures (all of these are flakes):
first failure seen in postsubmit is , part of ... preceeding many runs looked clean
I verified the delete options as seen by the server are identical before and after , so I don't think it is a functional change related to that PR
notably, many other tests in this suite started flaking on 3/7 and after ()
figured it out, dropped the detection of empty deletion options which would trigger the test to wait for all pods to be observed by the GC controller


Question: Alias            kube_containers Tag              kube_<namespace_name>_<pod_name>_<container_name> Tag_Regex        (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)- <del> Path             /var/log/containers/*.log </del> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> DB               flb_kube.db # Settings from fluentd missing here. # tag reform.* # format json # time_key time # time_format %Y-%m-%dT%H:%M:%S.%NZ </del> <ins> Path             C:varlogcontainers*.log DB               /var/run/google-fluentbit/pos-files/flb_kube.db [FILTER] Name         parser Match        kube_* Key_Name     log Reserve_Data True Parser       docker Parser       containerd </ins> # Example: <del> # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537] </del> <ins> # I0928 03:15:50.440223    4880 main.go:51] Starting CSI-Proxy Server ... </ins> [INPUT] Name             tail <del> Alias            kubelet Tag              kubelet #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            csi-proxy Tag              csi-proxy </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path /etc/kubernetes/logs/kubelet.log DB               /etc/kubernetes/logs/gcp-kubelet.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/csi-proxy.log DB               /var/run/google-fluentbit/pos-files/csi-proxy.db Multiline        On Parser_Firstline glog </ins> <del> # Example: # I0928 03:15:50.440223    4880 main.go:51] Starting CSI-Proxy Server ... </del> <ins> # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed </ins> [INPUT] Name             tail <del> Alias            csi-proxy Tag              csi-proxy #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            kube-proxy Tag              kube-proxy </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path             /etc/kubernetes/logs/csi-proxy.log DB               /etc/kubernetes/logs/gcp-csi-proxy.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/kube-proxy.log DB               /var/run/google-fluentbit/pos-files/kube-proxy.db Multiline        On Parser_Firstline glog </ins> # Example: # time="2019-12-10T21:27:59.836946700Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1 [INPUT] Name             tail Alias            container-runtime <del> Tag container-runtime #Multiline        on #Multiline_Flush  5 </del> <ins> Tag              container-runtime </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 Path             /etc/kubernetes/logs/containerd.log <del> DB               /etc/kubernetes/logs/gcp-containerd.log.pos # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ </del> <ins> DB               /var/run/google-fluentbit/pos-files/container-runtime.db # TODO: Add custom parser for containerd logs once format is settled. </ins> <del> # missing from fluentbit #   time_format %m%d %H:%M:%S.%N # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed </del> <ins> # Example: # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537] </ins> [INPUT] Name             tail <del> Alias            kube-proxy Tag              kube-proxy #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            kubelet Tag              kubelet </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path             /etc/kubernetes/logs/kube-proxy.log DB               /etc/kubernetes/logs/gcp-kubeproxy.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/kubelet.log DB               /var/run/google-fluentbit/pos-files/kubelet.db Multiline        On Parser_Firstline glog </ins> [FILTER] Name        modify Match       * Hard_rename log message <del> # [OUTPUT] #    Name        http #    Match       * #    Host        127.0.0.1 #    Port        2021 #    URI         /logs #    header_tag  FLUENT-TAG #    Format      msgpack #    Retry_Limit 2 </del> <ins> [FILTER] Name         parser Match        kube_* Key_Name     message Reserve_Data True Parser       glog Parser       json </ins> [OUTPUT] <del> name  stackdriver match * </del> <ins> Name        http Match       * Host        127.0.0.1 Port        2021 URI         /logs header_tag  FLUENT-TAG Format      msgpack Retry_Limit 2 </ins> '@ # Fluentbit parsers config file $PARSERS_CONFIG = @' [PARSER] Name        docker Format      json

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


Question: } // NewController constructs and returns a controller <del> func NewController(clock clock.Clock, client clientset.Interface, holderIdentity string, leaseDurationSeconds int32, onRepeatedHeartbeatFailure func()) Controller { </del> <ins> func NewController(clock clock.Clock, client clientset.Interface, holderIdentity string, leaseDurationSeconds int32, nodeStatusUpdateFrequency time.Duration, onRepeatedHeartbeatFailure func()) Controller { </ins> var leaseClient coordclientset.LeaseInterface if client != nil { leaseClient = client.CoordinationV1().Leases(corev1.NamespaceNodeLease) } <ins> renewInterval := defaultRenewInterval // Users are able to decrease the timeout after which nodes are being // marked as "Ready: Unknown" by NodeLifecycleController to values // smaller than defaultRenewInterval. Until the knob to configure // lease renew interval is exposed to user, we temporarily decrease // renewInterval based on the NodeStatusUpdateFrequency. if renewInterval > nodeStatusUpdateFrequency { renewInterval = nodeStatusUpdateFrequency } </ins> return &controller{ client:                     client, leaseClient:                leaseClient,

What happened: The node lease feature becomes beta and enables by default in v1.14. After upgrading our cluster to v1.14, we found some nodes become not ready occasionally and later become ready again after a few seconds. In order to have a quick failure detection and recovery, our kubelet has a flag setting and controller-manager has another flag setting . If we use node lease feature, the kubelet doesn't report it's status as frequently as specified by flag , and use node lease mechanism to report that node is alive. the default interval at which node lease renew is hardcoded 10s, but it can't be exactly 10s and may be a little bit more than 10s because of some network latency. when controller-manager doesn't receive messages from kubelet in every 10s in our case, it will mark this node as unreachable and not ready state, but in the next reconcile cycle, it receives the lease update message and mark this node as reachable again, so the node status is flapping between ready and not ready state. This seems a breaking change that setting kubelet's lease renewing interval to a hardcoded 10s, even though our node status update frequency has a little bit small value, node lease should align kubelet's previous behavior and renew as frequently as to reporting alive message, otherwise controller-manager will mark this node as unreachable. What you expected to happen: So we should not merely set node lease renewing interval to 10s, maybe depending on the has a more heuristic value. How to reproduce it (as minimally and precisely as possible): a v1.14 kubernetes cluster set flag to kubelet and to controller-manager. wait a few minute, 'll find some log regarding in controller-manager's log. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /kind bug /assign
smells like a regression WDYT?
cc
- not a regression - I think the only issue is that we may be missing some documentation, but other than that I think everything looks fine.

So how it works now is: There are two fields in Kubelet Config: (a) NodeStatusUpdateFrequency: (b) NodeStatusReportFrequency The comments there are pretty detailed and you should read them, but in short the first one roughly translates to how frequenty the lease object is send and the second how frequenty node status is updated (assuming nothing changed in the meantime). IIRC, there was a decision to not expose it via a flag, because flags in kubelet are deprecated and we're moving to config. So you can actually mitigate the problem by setting the second field in the config I mentioned above to some smaller value (like 5s) and then you will get Node Status updates every 5s as you want. So it's possible to achieve what you want without any code changes. That said, enabling for changing the default lease duration to something larger than 10s, sounds reasonable.
How about making the default lease duration 12s (or 14s) ? I can send out a PR once a new default is agreed upon.
thanks for comments. But what if we both want node lease feature enables (as this is an indeed helpful feature) and heartbeat with apiserver within some smaller interval (like 5s) via node lease mechanism?
IMHO, we should add a filed in Kubelet Config to let user tune the node lease renewing interval, WDYT?
That sounds like the cleanest option to me (rather than some playing with existing params). - who is the owner of the feature on Node side.


Question: "encoding/json" "fmt" "io" <del> "io/ioutil" </del> "strings" "unicode" jsonutil "k8s.io/apimachinery/pkg/util/json" <del> "k8s.io/klog/v2" </del> "sigs.k8s.io/yaml" )

This bug was privately disclosed and told to open here. What happened: NewYAMLOrJSONDecoder from has an index out of bounds issue which can cause callers to panic. Download attached reproducer file Run golang test: When run with v0.19.3: Additionally, this can be seen in kubectl directly (below is kubectl 1.20.0): Environment: Kubernetes version (use ): Verified with 1.19, 1.20
/sig api-machinery
This piece attempts to find line+column position of a json syntax error. To do that it assumes that full input is available which is not true: is : is only the first data chunk: The subject happens e.g. when syntax error offset is past buffer size and Test to reproduce the panic: I am not sure it is easily possible to find line+column of json syntax error as full input may not be available. Maybe it should only attempt to find line+column if offset is within the first chunk (i.e. below buffer size).
Line number calculation in and updated in
do you want to submit a patch to with a defensive check to avoid the panic ( compare and )
I have raised a discussion on having line number in the . The consensus there is that it should be possible to count lines with a full input at hand and it does not worth to have it in the core library due to performance impact and maintenance costs. The problem of here is that it attempts to find out line number without having full input at hand. I think the defensive patch is not the right approach because line calculation is incorrect anyway even when there is no panic: begins at a value with a syntax error, i.e. in a stream of two values and syntax error in the second value it would contain only the second value and thus new lines in the first value in the stream would not be counted. I think the following options are there: to stream taken from . This is straightforward but changes interface. I can not find any code that uses field, it is only used in the error message that would change: line number only if offset is within the length of the first data chunk (). I am not sure what would be the unknown line number then, perhaps. line number properly during stream decoding, see This has some performance penalty as line number is calculated in any case even if there is no syntax error, i.e.

in the majority of the cases.
cc /triage accepted
This is probably acceptable.
I have opened a PR
Thank you for support.


Question: # Start a single instance of hazelcast and set environment variables "DNS_DOMAIN=cluster" and "POD_NAMESPACE=default" in the container. kubectl run hazelcast --image=hazelcast --env="DNS_DOMAIN=cluster" --env="POD_NAMESPACE=default" <ins> # Start a single instance of hazelcast and set labels "app=hazelcast" and "env=prod" in the container. kubectl run hazelcast --image=nginx --labels="app=hazelcast,env=prod" </ins> # Start a replicated instance of nginx. kubectl run nginx --image=nginx --replicas=5

If you run a command like: $ kubectl run --image=nginx --labels app=web --labels env=prod web It only uses the latest declaration of and drops the earlier options specified. Especially because the --help documentation on does not specify the format, this is impairing the user experience. I looked around a few other CLIs to see how they handle this. curl, for example, uses to additively collect headers (instead of a comma-separated list), therefore each value specified is used. "docker run" command ignores the earlier --name argument when you specify another one. I am wondering if we can easily detect non-additive/overwriting args specified multiple times in an invocation. /kind bug /sig cli
I think this is reasonable for improving cli tool user experience. For different commands, it may different in the options, but for each of them we can add warning messages at least for the end user to realize what had happened in the backend, especially for the ignored staff
/sig cli
why use for flag instead of ? i don't think should overwrite
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: 500, /* cpuLimit */ 200, /* memLimit */ f) <del> common.CreateCPUHorizontalPodAutoscaler( </del> <ins> t.hpa = common.CreateCPUHorizontalPodAutoscaler( </ins> t.rc, 20, /* targetCPUUtilizationPercent */ 1,  /* minPods */

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: // BuildInsecureHandlerChain sets up the server to listen to http. Should be removed. func BuildInsecureHandlerChain(apiHandler http.Handler, c *server.Config) http.Handler { handler := apiHandler <ins> handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc) </ins> handler = genericapifilters.WithAudit(handler, c.AuditBackend, c.AuditPolicyChecker, c.LongRunningFunc) handler = genericapifilters.WithAuthentication(handler, server.InsecureSuperuser{}, nil, nil) handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true") handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout) <del> handler = genericfilters.WithMaxInFlightLimit(handler, c.MaxRequestsInFlight, c.MaxMutatingRequestsInFlight, c.LongRunningFunc) </del> handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup) handler = genericapifilters.WithRequestInfo(handler, server.NewRequestInfoResolver(c)) handler = genericapifilters.WithCacheControl(handler)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Working on a generalization of the max-in-flight filter, I noticed it is positioned inconsistently in the secure vs insecure handler chains. At I see it is very late in the secure handler chain, just before authorization. At I see it comes much earlier, preceding audit, authentication, and timeout (unlike the secured chain). NB: by "early" and "late" here I refer to the order in which the filters run, which is the opposite of the order in which they are constructed. What you expected to happen: I expected the two handler chains to put the filters they have in common in the same order. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc


Question: "//pkg/client/informers/informers_generated/externalversions:go_default_library", "//pkg/controller:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", </ins> ], )

Seen in happy to send a fix but didn't know if this might have been address with your current refactors. Output:
This looks new. I'd happily review a fix. Thanks.
Dupe of


Question: "k8s.io/kubernetes/pkg/features" internalapi "k8s.io/kubernetes/pkg/kubelet/apis/cri" kubeletconfiginternal "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" <del> kubeletconfigv1alpha1 "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1alpha1" </del> "k8s.io/kubernetes/pkg/kubelet/cadvisor" kubeletcertificate "k8s.io/kubernetes/pkg/kubelet/certificate" "k8s.io/kubernetes/pkg/kubelet/cm"

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: if err != nil { errors = append(errors, err) } <del> default: </del> } }

Running : cpu usage goes to 2 cores @ 100% What you expected to happen: Way lower cpu usage. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ):Kernel (e.g. ): Linux felix-Latitude-7490 5.3.0-62-generic ~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x8664 x8664 x86_64 GNU/Linux
Hey I just did not understood, the cpu goes to 200% in the node or in the place where kubectl is running? Thanks!
/priority awaiting-more-evidence /sig cli
On my laptop running kubectl
Same issue on mac: kubectl version Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} PID COMMAND %CPU TIME #TH #WQ #PORTS MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPUME %CPUOTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEINS IDLEW POWER INSTRS CYCLES USER #MREGS RPRVT VPRVT VSIZE KPRVT KSHRD kubectl 100.5 03:30.41 32/1 0 43 16M 0B 0B 1322 running *0[1] 0. 0. 274 36 16 + 117 + 3730 266+ 100.5 costel.lupoaie N/A N/A N/A N/A N/A N/A I noticed the problem after upgrade from 1.15.

Maybe it't not kubectl itself but the way it communicates with external auth provider like IAM in my case.
oh I see. Thanks I've no access to cloud providers right now to test, but as you've provisioned in EKS using IAM and is using GKE it MIGHT be a thing. Will wait to see if someone else have access to cloud providers and can test this :)
Maybe I am wrong and it's just some busy waiting. In my case I noticed it when draining nodes with many pods that cannot immediately be stopped because it would break the deployments maxUnavailable limits, so kubectl has to wait until it can stop all the pods on the draining node.
In my case also this was triggered when pods couldn't be evicted immediately because of a violation of pdb.
does not lead to high cpu usage in my case, so I suspect it has nothing to do with IAM/GKE auth, but with evictions failing because of pdb (but I am not experienced enough to know kubectl works in the background).
I can confirm 100% (not 200%) CPU core usage for each process. Currently I'm draining four nodes in parallel (4x ), all four are waiting due to PDB's, and all are at 100% CPU most of the time. I'll try with a newer kubectl next time.
OK, so it seems to be related to PDB and Drain. I'll try to take a look into it, try to reproduce with the last supported clients. Thank you for reporting! /assign
Thanks for looking into it! And my apologies, I forgot to comment on this before: Our cluster is deployed on bare metal on premise, so it's probably not EKS og GKE related.
Is that possible you try draining with "disable-eviction" just to make sure PDB or eviction control is causing this? Thanks!
I found a newer kubectl lying around, so it's now v1.18.8 (1.17 didn't include it seems) I couldn't test on the same cluster, but I just drained a host from our staging env with a similar amount of pods to what we run in prod (~85 vs ~100), and I think you're right. CPU usage certainly didn't go above 20% at any point on any core, and that's basically background noise on this host.

So my theory here is that kubectl keeps waiting for the node to drain and this is causing a high CPU usage. I'm also creating a cluster here to reproduce the behavior, and see what part of the code is this being caused
Confirmed here :) Even in last kubectl v1.19.2 the problem happens. It seems also it's putting some pressure in API Server because of the amount of nodes vs PDB. disable-eviction is a "workaround" not desired if using PDB /remove-priority awaiting-more-evidence /priority important-soon
I'm hitting my head in the wall here trying to figure out why the goroutine causes the peak of CPU usage, so anyone with better skills then me in Go, feel free to join :) is the function taking the whole CPU apparently Will keep digging what part of this function is taking the whole CPU. When removing the and making this 'serial' it seems to keep the CPU in low usage but I don't know if this is the desired scenario, as draining a node could take much longer since other Pods may not be using PDB. EDIT: but it also seems that the drain does not goes further because I forgot to change the return/break stuff inside the loop. So I'm thinking tomorrow maybe create a new 'queue' of the 'failed because of PDB' Pods, and deal with them in a separate way.


Question: --filter='+ /vendor/'  --filter='+ /staging/***/Godeps/**'  --filter='+ /_output/dockerized/bin/**'  <ins> --filter='- /_output/dockerized/go/**'  </ins> --filter='+ zz_generated.*'  --filter='+ generated.proto'  --filter='+ *.pb.go'

While debugging I discovered that setting causes @ (recent master branch commit) to fail on cleanup at the end. This doesn't happen if you don't set . It's not immediately apparent why those are related. This make target should not fail on a clean branch with no diff. set GOTOOLCHAIN to some valid go version and then run /sig testing release /triage accepted n/a Linux. ~Debian. But this is happening inside our build container. n/a docker 20.10.21 n/a
I would guess: we wind up downloading go underneath the tempdir we setup for the script and then there's some non-writeable go toolchain download cache there, or something like that.
That is indeed the case. Go toolchains are downloaded into and the folder is write protected. I will send in a fix. /assign


Question: # https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/delete_cluster.go#L38 #: staging/src/k8s.io/kubectl/pkg/cmd/config/delete_cluster.go:42 msgid "Delete the specified cluster from the kubeconfig" <del> msgstr "指定したコンテキストをkubeconfigから削除する" </del> <ins> msgstr "指定したクラスターをkubeconfigから削除する" </ins> # https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/config/delete_context.go#L38 #: staging/src/k8s.io/kubectl/pkg/cmd/config/delete_context.go:42

There is a minor translation issue about in Japanese. Output of and are completely same. means cluster in japanese, so it must be
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig docs


Question: ### Getting VMs <del> 1. You need a Google Cloud Platform account with billing enabled. Visit [http://cloud.google.com/console](http://cloud.google.com/console) for more details. 2. Make sure you can start up a GCE VM.  At least make sure you can do the [Create an instance](https://developers.google.com/compute/docs/quickstart#addvm) part of the GCE Quickstart. 3. Make sure you can ssh into the VM without interactive prompts. </del> <ins> 1. You need a Google Cloud Platform account with billing enabled. Visit the [Google Developers Console](http://cloud.google.com/console) for more details. 2. Make sure you can start up a GCE VM from the command line.  At least make sure you can do the [Create an instance](https://cloud.google.com/compute/docs/quickstart#create_an_instance) part of the GCE Quickstart. 3. Make sure you can ssh into the VM without interactive prompts.  See the [Log in to the instance](https://cloud.google.com/compute/docs/quickstart#ssh) part of the GCE Quickstart. </ins> * Your GCE SSH key must either have no passcode or you need to be using `ssh-agent`. * Ensure the GCE firewall isn't blocking port 22 to your VMs.  By default, this should work but if you have edited firewall rules or created a new non-default network, you'll need to expose it: `gcloud compute firewall-rules create --network=<network-name> --description "SSH allowed from anywhere" --allow tcp:22 default-ssh` <del> 4. You need to have the Google Cloud Storage API, and the Google Cloud Storage JSON API enabled. This can be done in the Google Cloud Console. </del> <ins> 4. You need to have the Google Cloud Storage API, and the Google Cloud Storage JSON API enabled. It is activated by default for new projects. Otherwise, it can be done in the Google Cloud Console.  See the [Google Cloud Storage JSON API Overview](https://cloud.google.com/storage/docs/json_api/) for more details. </ins> ### Prerequisites for your workstation <del> 1. Be running a Linux or Mac OS X. </del> <ins> 1. You must be running Linux or Mac OS X on your workstation. </ins> 2. You must have the [Google Cloud SDK](https://developers.google.com/cloud/sdk/) installed.  This will get you `gcloud` and `gsutil`. <del> 3. Ensure that your `gcloud` components are up-to-date by running `gcloud components update`. 4. If you want to build your own release, you need to have [Docker installed](https://docs.docker.com/installation/).  On Mac OS X you can use [boot2docker](http://boot2docker.io/). (see also: https://docs.docker.com/installation/mac/) 5. Get or build a [binary release](binary_release.md) </del> <ins> 3. Install `gcloud preview`: run `gcloud components update preview` to make sure it is. 4. Ensure that your other `gcloud` components are up-to-date by running `gcloud components update`. 5. If you want to build your own release, you need to have [Docker installed](https://docs.docker.com/installation/).  On Mac OS X you can use [boot2docker](http://boot2docker.io/). (see also: https://docs.docker.com/installation/mac/) 6. Get or build a [binary release](binary_release.md) of Kubernetes. </ins> ### Starting a Cluster <ins> Change into the `kubernetes` directory in which you have the binary release, and then do </ins> ```bash cluster/kube-up.sh ``` <ins> (If it fails, do `cluster/kube-down.sh` to clean up before trying again; otherwise, you'll get errors about resources that already exist.) </ins> The script above relies on Google Storage to stage the Kubernetes release. It then will start (by default) a single master VM along with 4 worker VMs.  You can tweak some of these parameters by editing `cluster/gce/config-default.sh`

I'm a recent user of Kubernetes, and I struggled a bit to get it up and running on GCE. It looks to me like the Getting started on Google Compute Engine]() doc is a bit out of date. I've updated the doc with what I think it should look like, and I'll link the PR here. The main issues I found are: The "Getting VMs" section could be clearer, and some of the docs linked to are out of date. I believe is now required for . Failure is messy with , so a note to run in case of failure before re-running would be helpful.
Isaac - thanks for these fixes! (we've been a bit negligent about making sure all of our documentation stays up to date as the underlying source/examples have been churning so quickly).


Question: rbacv1 "k8s.io/api/rbac/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/runtime/schema" types "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid"

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? It was picked up in APISnoop that an untested endpoint was promoted from Beta to GA. PR ingress: promote CRUD API tests for v1 to conformance covered the Ingress life cycle tests, but readNetworkingV1NamespacedIngressStatus was missed. According to this APIsnoop readNetworkingV1NamespacedIngressStatus endpoint is untested. ginkgo.By("get /status") ingResource := schema.GroupVersionResource{Group: "", Version: ingVersion, Resource: "ingresses"} gottenStatus, err := f.DynamicClient.Resource(ingResource).Namespace(ns).Get(context.TODO(), createdIngress.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err) statusUID, _, err := unstructured.NestedFieldCopy(gottenStatus.Object, "metadata", "uid") framework.ExpectNoError(err) framework.ExpectEqual(string(createdIngress.UID), statusUID, fmt.Sprintf("createdIngress.UID: %v expected to match statusUID: %v ", createdIngress.UID, statusUID)) ''' List endpoints hit by the test: readNetworkingV1NamespacedIngressStatus endpoint If this test get merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Duplicate of
/reopen Merged /assign Ready for creation of Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged /assign Ready for creation of Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change was a bug fix in a Merged conformance test. The correction should not require Promotion. /close
Closing this issue. <detailsIn response to : Code change was a bug fix in a Merged conformance test. The correction should not require Promotion. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details