Question: "unicode" "unicode/utf8" <ins> "github.com/google/go-cmp/cmp" </ins> v1 "k8s.io/api/core/v1" apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/resource"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --pod with nodeSeletor edit pod, delete the nodeSeletor field report error with: ! The errors message should displayed as: Because the nodeSeletor is been deleted with this edit. Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle rotten


Question: scheduleAttempts, e2eSchedulingLatency, SchedulingAlgorithmLatency, <del> BindingLatency, SchedulingAlgorithmPreemptionEvaluationDuration, </del> <ins> DeprecatedBindingLatency, DeprecatedSchedulingAlgorithmPreemptionEvaluationDuration, </ins> PreemptionVictims, PreemptionAttempts, pendingPods,

Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. This can be done by setting in each of the two metrics opts. Changing the name of the metric variable to be prefixed with "Deprecated" is useful as well. /help /sig scheduling
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/kind deprecation


Question: } } <ins> func TestCreatePrivilegedContainerForOpenStack(t *testing.T) { // Creates a Master Configuration with OpenStack cloud provider var staticPodNames = []string{ kubeadmconstants.KubeAPIServer, kubeadmconstants.KubeControllerManager, } var assertions = []struct { cloudProvider     string privilegedPods    bool expectedPrivilege bool }{ { cloudProvider:     "", expectedPrivilege: false, }, { cloudProvider:     "aws", expectedPrivilege: false, }, { cloudProvider:     "openstack", privilegedPods:    true, expectedPrivilege: true, }, } for _, assertion := range assertions { cfg := &kubeadmapi.MasterConfiguration{ KubernetesVersion: "v1.9.0", CloudProvider:     assertion.cloudProvider, PrivilegedPods:    assertion.privilegedPods, } k8sVersion, _ := version.ParseSemantic(cfg.KubernetesVersion) specs := GetStaticPodSpecs(cfg, k8sVersion) for _, podname := range staticPodNames { spec, _ := specs[podname] sc := spec.Spec.Containers[0].SecurityContext if assertion.expectedPrivilege == true { if sc == nil || sc.Privileged == nil || *sc.Privileged == false { t.Errorf("GetStaticPodSpecs did not enable privileged containers in %s pod for provider %s", podname, assertion.cloudProvider) } } else { if sc != nil && sc.Privileged != nil && *sc.Privileged == true { t.Errorf("GetStaticPodSpecs enabled privileged containers in %s pod for provider %s", podname, assertion.cloudProvider) } } } } } </ins> func TestGetAPIServerCommand(t *testing.T) { var tests = []struct { cfg      *kubeadmapi.MasterConfiguration

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: return mountedVolumes, nil } <ins> // podVolumesSubpathsDirExists returns true if the pod volume-subpaths directory for // a given pod exists func (kl *Kubelet) podVolumeSubpathsDirExists(podUID types.UID) (bool, error) { podVolDir := kl.getPodVolumeSubpathsDir(podUID) if pathExists, pathErr := volumeutil.PathExists(podVolDir); pathErr != nil { return true, fmt.Errorf("Error checking if path %q exists: %v", podVolDir, pathErr) } else if !pathExists { return false, nil } return true, nil } </ins> // GetVersionInfo returns information about the version of cAdvisor in use. func (kl *Kubelet) GetVersionInfo() (*cadvisorapiv1.VersionInfo, error) { return kl.cadvisor.VersionInfo()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" <ins> clientscheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image"

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 1 Aug 2022 [x] test promotion pr : According to this APIsnoop query, there is still some remaining NamespaceStatus endpoints which are untested. readCoreV1NamespaceStatuspatchCoreV1NamespaceStatus Using a number of existing e2e test practices a new has been created for 2 namespace status endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fakeClaimWatch := watch.NewFake() client.PrependWatchReactor("persistentvolumeclaims", core.DefaultWatchReactor(fakeClaimWatch, nil)) client.PrependWatchReactor("storageclasses", core.DefaultWatchReactor(watch.NewFake(), nil)) <ins> client.PrependWatchReactor("nodes", core.DefaultWatchReactor(watch.NewFake(), nil)) client.PrependWatchReactor("pods", core.DefaultWatchReactor(watch.NewFake(), nil)) </ins> informers := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) ctrl, err := newTestController(client, informers, true)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit TestControllerSync /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen I still see the flake in recently.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fix:


Question: default: return nil, fmt.Errorf("unsupported algorithm source: %v", source) } <del> metrics.Register() </del> // Additional tweaks to the config produced by the configurator. sched.Recorder = recorder sched.DisablePreemption = options.disablePreemption

What happened: two scheduler metrics, and , was not recorded in . scheduler has some code for this already. It seems to be a bug. What you expected to happen: and was recorded How to reproduce it (as minimally and precisely as possible): /king bug /sig scheduling
/assign
What do you expect for ? And I think pending pods are available in metrics, but not from scheduler. You can get it from pods info metrics.
actually, scheduler has some code already for recording these metrics. But I found a bug in these code. I will post my PR to fix this issue soon. // I updated the issue description for clarifying.


Question: // NOTE: The $Format strings are replaced during 'git archive' thanks to the // companion .gitattributes file containing 'export-subst' in this same // directory.  See also https://git-scm.com/docs/gitattributes <del> gitVersion   string = "v0.0.0-master+$Format:%h$" </del> <ins> gitVersion   string = "v0.0.0-master+$Format:%H$" </ins> gitCommit    string = "$Format:%H$" // sha1 from git, output of $(git rev-parse HEAD) gitTreeState string = ""            // state of git tree, either "clean" or "dirty"

Ref. includes: // NOTE: The $Format strings are replaced during 'git archive' thanks to the // companion .gitattributes file containing 'export-subst' in this same // directory. See also https://git- gitVersion string = "v0.0.0-master+" This directive, activated by and the  containing the line , results in file contents served by GitHub Archives that can change in unexpected ways. For example, the length of the commit SHA can expand over time as there are collisions. This non-determinism can lead to surprises, such as , where a client-go dependency resulted in unexpectedly different tar hashes, which was detected when a new version of the ko binary was being to Homebrew. tl;dr: though the client-go dependency itself didn't change for ko, later runs of (via GitHub Archives) resulted in different contents for the string to avoid a new collision for the previous prefix length, which cascaded into different content SHAs and shady-looking binary SHAs. I'm not sure, but after reading docs I believe you could naively solve this by replacing with so the string includes the full commit hash instead of the abbreviated one. I have no idea if anything depends on being a shorter string, but presumably things are fine with it having been a variable-length string so . Given enough time/collisions, I suppose theoretically will slow grow to be equal to , but I don't think that will happen during any of our lifetimes cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the report. Moving to kubernetes/kubernetes where this source comes from. There's not a lot we can do for past releases, but I'd be in favor of switching to %H going forward. Reproducibility of source archives is important.
/sig release
/kind bug
Thanks for taking this on !!
since we don't actually use to full-fill the commit info as version, should we just delete that file? containerd a issue about checksum mismatch due to this.

and before that it was and before that , so everywhere
I have created to stop including the files in k8s libraries. For anyone vendoring these libraries in the future, you should no longer be affected by this issue.


Question: embed = [":go_default_library"], deps = [ "//vendor/github.com/Azure/azure-sdk-for-go/services/containerregistry/mgmt/2019-05-01/containerregistry:go_default_library", <ins> "//vendor/github.com/Azure/go-autorest/autorest/azure:go_default_library", </ins> "//vendor/github.com/Azure/go-autorest/autorest/to:go_default_library", ], )

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: }, }, expectedUpdateCalls: []fakecloud.UpdateBalancerCall{ <ins> {Service: etpLocalservice1, Hosts: []*v1.Node{node1, node3}}, {Service: etpLocalservice2, Hosts: []*v1.Node{node1, node3}}, </ins> {Service: service3, Hosts: []*v1.Node{node1, node3}}, }, }, {

In 1.26 the service controller in the Kubernetes cloud controller manage (KCCM) introduced a new mechanism for syncing load balancers based on predicates. The predicates applied will depend on "service classes", specifically if it's or , see: This implementation is not well suited for internal GCP load balancers which use InstanceGroups. from Google, feel free to correct me: any cluster has all nodes grouped together into an instance group. Service of have individual GCP load balancers, but these load balancers end up referencing the same instance group (which is the cluster's instance group). This means that the inclusion/exclusion predicates for any service, will trickle down and impact the cluster's global instance group, thereby impacting other services which don't necessarily apply the same set of predicates. Ex: services exclude nodes, whereas do not. A Node going will therefore be removed from all backends for services, however on GCP with load balancer solution based on InstanceGroups: this means that the same node will be removed from all backends for services too. This is NOT a problem on = 1.27 using the feature , because all services on 1.27 using that feature gate will have the same set of predicates. Node updates requiring the removal of the instance from load balancers belonging to services of should not impact other services of . Bring up a cluster using "Internal load balancers" on GCP. Define two services, one being and the other . Have one node go . The instance corresponding to that Node might be removed from the InstanceGroup backend of the service. This will depend on the order the services are synced, if the service is synced last then the instance will be removed. This is actually not really that bad, because the only different predicate between both is the readiness condition - and a node should go back to if all is fine, and finally be included. But I wanted to raise this for issue for completeness sake so that it is known. <details</details<detailsGCP </details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/triage accepted


Question: * extensions/v1beta1.Jobs is deprecated, use batch/v1.Job instead ([[#36355](https://github.com/kubernetes/kubernetes/pull/36355)](https://github.com/kubernetes/kubernetes/pull/36355), [[@soltysh](https://github.com/soltysh)](https://github.com/soltysh)) * The kubelet --reconcile-cdir flag is deprecated because it has no function anymore. ([[#35523](https://github.com/kubernetes/kubernetes/pull/35523)](https://github.com/kubernetes/kubernetes/pull/35523), [[@luxas](https://github.com/luxas)](https://github.com/luxas)) * Notice of deprecation for recycler [[#36760](https://github.com/kubernetes/kubernetes/pull/36760)](https://github.com/kubernetes/kubernetes/pull/36760) <ins> * The init-container (pod.beta.kubernetes.io/init-containers) annotations used to accept capitalized field names that could be accidently generated by the k8s.io/kubernetes/pkg/api package. Using an upper case field name will now return an error and all users should use the versioned API types from `pkg/api/v1` when serializing from Golang. </ins> ## Action Required Before Upgrading

What? The following pod yaml works on 1.4 cluster: However, creating it against 1.5.1 or master branch gives the following error:
cc
Your annotations are wrong - you're using upper case field names instead of lower case field names - i.e. "Env" is not correct, but "env" is.
If you serialized that from a go object, it means you are using the internal type (pkg/api) instead of the versioned type (pkg/api/v1), which was always subtly broken. We removed the JSON fields to prevent accidents like this.
Since this used to work and no longer does can we add it to the ? I guess the entry would say:
Yeah - I'm curious as to how it worked before, because we invoked a versioned decode from external (or should have) and either way only one deserialization should have been possible (lower or upper, but not both).
Can we reopen this? Still doesn't work. yaml: On 1.5: Successful on 1.4
So it seems that it also accidentally accepts empty apiVersion in 1.4?
There is some kind of defaulting mechanism, I believe. For example, when you create a pod, you don't specify "apiVersion".
Downward api should be defaulted during a defaulting pass. There's an existing bug on that. For 1.5 you have to fill in the defaults yourself
Thanks for answering. Is there any issue created against it? We can help fix the issue if needed.
Was reported in and fixed in the linked PRs. , Hongchao Deng <:
I actually tried the latest master today and no luck. The issue only tested imagePullPolicy. That might be not enough.
It's possible that something else is interfering with defaulting. Please report that on the other issue
You mean create an new issue?
I believe that means comment on the closed issue: , correct?
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> "//vendor/k8s.io/apimachinery/pkg/util/validation/field:go_default_library", </del> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/yaml:go_default_library", "//vendor/k8s.io/client-go/kubernetes:go_default_library",

To read a pod manifest file from disk (or from the http source), kubelet execute the following code to Decode a on-disk v1 pod manifest to an internal pod object (the defaults are applied in this process). Validate the internal pod. Convert the internal pod back to a v1 pod. The downstream pod cache (which gets v1 pods from different sources) performs similar operations: Convert the v1 pod to an internal pod. Validate the internal pod. One of two conversion/validation should be removed, and some of them should be simplified.
/cc for suggestions on defaulting and conversion.
If no one is working on this, I will take it.


Question: # Default registry, arch and tag. This can be overwritten by arguments to make PLATFORM?=linux ARCH?=amd64 <del> TAG?=1.6 </del> <ins> TAG?=1.7 </ins> REGISTRY?=gcr.io/google_containers GOLANG_VERSION=1.6

Now that merged, we need to build a new kube-dns release.
I just put in a PR yesterday to fix the missing changelog entry for kube-dns:1.6


Question: clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod"

[X] Test PR: kuberenetes/kubernetes 17 June 2021 [x] Two weeks soak end date: 1 Jul 2021 [X] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track stateful set events Create a stateful set with a static label. Confirm that the pods are running. Get the stateful set status. Parse the response and confirm that the stateful set status conditions can be listed. Update the stateful set status. Confirm via the watch that the status has been updated. Patch the stateful set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for stateful set lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> See [e2e-tests](https://git.k8s.io/community/contributors/devel/sig-testing/e2e-tests.md) </del> <ins> # test/e2e This is home to e2e tests used for presubmit, periodic, and postsubmit jobs. Some of these jobs are merge-blocking, some are release-blocking. ## e2e test ownership All e2e tests must adhere to the following policies: - the test must be owned by one and only one SIG - the test must live in/underneath a sig-owned package matching pattern: `test/e2e/[{subpath}/]{sig}/...`, e.g. - `test/e2e/auth` - all tests owned by sig-`auth` - `test/e2e/common/storage` - all tests `common` to cluster-level and node-level e2e tests, owned by sig-`node` - `test/e2e/upgrade/apps` - all tests used in `upgrade` testing, owned by sig-`apps` - each sig-owned package should have an OWNERS file defining relevant approvers and labels for the owning sig, e.g. ```yaml # test/e2e/node/OWNERS # See the OWNERS docs at https://go.k8s.io/owners approvers: - alice - bob - cynthia emeritus_approvers: - dave reviewers: - sig-node-reviewers labels: - sig/node ``` - packages that use `{subpath}` should have an `imports.go` file importing sig-owned packages (for ginkgo's benefit), e.g. ```golang // test/e2e/common/imports.go package common import ( // ensure these packages are scanned by ginkgo for e2e tests _ "k8s.io/kubernetes/test/e2e/common/network" _ "k8s.io/kubernetes/test/e2e/common/node" _ "k8s.io/kubernetes/test/e2e/common/storage" ) ``` - test ownership must be declared via a top-level SIGDescribe call defined in the sig-owned package, e.g. ```golang // test/e2e/lifecycle/framework.go package lifecycle import "github.com/onsi/ginkgo" // SIGDescribe annotates the test with the SIG label. func SIGDescribe(text string, body func()) bool { return ginkgo.Describe("[sig-cluster-lifecycle] "+text, body) } ``` ```golang // test/e2e/lifecycle/bootstrap/bootstrap_signer.go package bootstrap import ( "github.com/onsi/ginkgo" "k8s.io/kubernetes/test/e2e/lifecycle" ) var _ = lifecycle.SIGDescribe("[Feature:BootstrapTokens]", func() { /* ... */ ginkgo.It("should sign the new added bootstrap tokens", func() { /* ... */ }) /* etc */ }) ``` These polices are enforced: - via the merge-blocking presubmit job `pull-kubernetes-verify` - which ends up running `hack/verify-e2e-test-ownership.sh` - which can also be run via `make verify WHAT=e2e-test-ownership` ## more info See [kubernetes/community/.../e2e-tests.md](https://git.k8s.io/community/contributors/devel/sig-testing/e2e-tests.md) </ins>

Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests. However, not all tests have such a tag. We should: assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name EDIT: proposed path forward (pulled out of and ) rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and nodee2e add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( follow these conventions: top-level calls are required, nested calls are not allowed remove usage of , , , and in favor of nested calls are ok, nested calls are not enforce these conventions via some combination of: static analysis regex enforcement against names output during a run while we are here, is vestigial and can be removed (ref: ()) EDIT: Everything now conforms to policies defined above will close this out by adding a blocking test and docs to /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Many e2e tests today have a tag in their name, which lets us know which sig is responsible for fixing failing or flaky e2e tests.

assign tags to all e2e tests that don't yet have them enforce that no new e2e tests can be without a tag in their name /sig testing /area test for enforcement (FYI /sig architecture /area conformance some of these are conformance tests (FYI /wg reliability for helping determine appropriate sig ownership (FYI ) /sig release since this is relevant to figuring out who should fix release-blocking tests (FYI /priority important-soon /milestone v1.21 I think it's reasonable to aim to accomplish this prior to code freeze for v1.21 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Assigning an arbitrary lead from each SIG/WG mentioned, please assign to someone more appropriate if you don't have the bandwidth
I can't see anything sig-network related :thinking: there are a lot of NodeConformance things though ,
/assign for visibility - lots of these are node /sig node
/assign (for sig-node)
I've just sent out for autoscaling tests. It seems that most of the remaining tests are node-related. We should try to fix most of these to see if there is anything that is hidden between those.
/unassign
Sent to address rest of the tests. Echoing the review comment from I agree, we can remove prefix once all the tests are tagged. wdyt ? I'll raise a follow up to clean the prefix.
re: /assign (also maybe knows) at one time I thought Openshift was relying on to filter out upstream tests for certain jobs? I don't see that it's consistently applied some I'm assuming we could drop from our tests? (I said as much here: )
We synthetically apply that (if the source package is in ) so can be dropped. Also note in generally I am massively in favor of this change - I forced this to happen in openshift so some of the unassigned upstream tests got blanket assigned to equivalent teams (i.e. any test in that didn't have a sig had to get one synthetically ) so this will remove the need for it. Having owners just clarified everything, and then the ratcheting "you may not add one without" also ensured folks thought even a tiny bit about ownership.

Fleshing out what I suggested in and Can we agree to do the following? rearrange e2e tests into sig-owned packages, - e2e tests that run as part of nodee2e - e2e tests that run as part of e2e and node_e2e ensure each sig-owned package has: a package-local () an OWNERS files with sig members and ( remove usage of , , , and in favor of add that imports sub-packages so e2e binaries can continue to import to pick up test definitions () Enforcing test ownership then looks like: an e2e test must live in a sig-owned package (or subpackage) an e2e test must be defined by that package's PRs to that package will be assigned / auto-labeled to the appropriate SIG changing test ownership is a move from one package to another, requires approval from both the old SIG and new SIG Implications I don't have the foresight to see as blockers, can you think of any? we disallow the concept of e2e packages containing tests owned by multiple sigs some moves may require imports to change or things like helper functions and fixtures to get extracted Folks seemed to like the idea at the time but I'm seeing PRs move in different directions. How can I help get us on the same page?
+1 for that I think that the PR above as an example is an intermediate step (it's not a different direction, rather just one step instead of multiple at once).
There is one question that I have at this point - what do we do with nested Described, e.g.: this one: [there is KubeDescribe nested in SIGDescribe]
I think that having SIGDescribe twice in those cases doesn't seem to be the best option, and those cases seem to be significant percentage of remaining cases...
I don't have strong opinion here. Mostly curious why prefer the non-explicit variant? Is it to optimize for tests migrations?
There is one question that I have at this point - what do we do with nested Described I can't think of a strong reason to ban them, and I think it would be more trouble than it's worth to come up with alternatives for how they're used now.

I can't think of a compile-time way to enforce that "[sig-foo]" doesn't show up in their text, so I'm not sure it's worth keeping/enforcing use of wrappers (they definitely make error reporting worse) So something like: replace nested calls with follow these conventions: top-level calls are required, nested calls are not allowed nested calls are ok, nested calls are not enforcement of these conventions can be done via some combination of static analysis and regex check against the full names generated by a run
That makes sense to me. I will start looking into migrating existing tests to satisfy these rules in free cycle, but i'm leaving the enforcement to you :)
I do consider not having to modify test code to change ownership a benefit. But it's more that we'd have to change a lot more code to conform to than . Since we mostly already have the pattern in place, I'd rather stick to it.
Thinking of timeline (ref: ) This work can continue past Code Freeze on Mar 9, but that's a useful milestone to evaluate our progress I'll update the issue description as we get tests owned My goal is to get enforcement landed sometime after Code Freeze This work can't continue past Test Freeze on Mar 24
- also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests)
also one more thing to do would be to ensure that no tests are tagged with more then one SIG (which I think isn't true now for a few tests) Good point. There are tinier corner cases, but let's start with the big one. /sig storage /sig windows /assign There are a bunch of storage tests that look like the following: Which SIG should own these?
We make us of for many e2e tests that are intended to run on windows machines. My suggestion here is probably to have sig-storage own these tests as they reside under /test/e2e/storage/* and switch to using instead of in the test names. SIG-windows will always be happy to help debug issues here. - thought?
I think it makes sense for storage to own the windows storage tests. Using [Feature:Windows] seems reasonable although we'll have to update a number of test jobs.
Working on in-tree enforcement before I update description.

tl;dr the majority of tests listed in the description are still not tagged (with at least one SIG), waiting on SIG Node to sign off on usage of KubeDescribe has gone way down, but not at zero yet
is the biggest bit now (once I get feedback from SIG node, we will be almost there with both)
refreshing description based on script in
Regarding "multi-ownership", with windows test figured (the PR didn't merge yet, but hopefully will soon), the biggest question is now about the upgrade tests. - there are a number of apps related upgrade tests. Even though they are currently running on GCP, I suggest that we officially make them sig-apps tests (I'm sure sig-cloud-provider-gcp will be happy to help on infrastructural problems). Thoughts?
can you be more specific about which particular tests? I quickly checked and those are already marked as sig-apps. I don't have any objections as long as they are part of sig-apps territory but I'd like to know a concrete list before saying so :wink:
- sorry, I should have been more specific. Many of those tests are actually being run from here: and the tests there are marked as [sig-cloud-provider-gcp] As part of ensuring that each test has a one single sig ownership, I suggest making it explicit that all of those are sig-apps. Does that make more sense now?
most - yes, but I see a few exceptions: upgradeTest: ServiceUpgradeTest - seems like sig-network SecretUpgradeTest - looks like assigned to sig-storage/sig-apimachinery ConfigMapUpgradeTest - looks like assigned to sig-storage/sig-apimachinery HPAUpgradeTest - sig-autoscaling PersistentVolumeUpgradeTest - sig-storage AppArmorUpgradeTest - sig-auth (?) VolumeModeDowngradeTest - sig-storage gpuUpgradeTests - looks like sig-node/sig-scheduling kubeProxyUpgradeTests - both sig-network kubeProxyDowngradeTests - both sig-network serviceaccountAdmissionControllerMigrationTests - looks like sig-apimachinery the remaining less than half :+1: for sig-apps.
Sure - I didn't mean to put all of them to sig-apps. There is just the biggest group of apps related one so wanted to start with this group.
I'm just making sure :wink: Go for it and thx :100:
Hello!

The code freeze is starting March 9th, 2021 (about 1 week from now). As the Issue is tagged for 1.21, is it still planned for this release?
Yes this is planned for this release, and PRs related to this are not subject to code freeze. This will be completed by test freeze (see timeline here: )
- calling your script from head, I have: So there are just couple exceptions now (and a mess with upgrade tests :) )
- when you have time, there are couple PRs waiting for you now
Not reviewing on the weekend, but will check them out first thing next week. Also ok if someone else gets to them before me
Run in master. The is due to The are all from It seems it is done.
The two opened PRs + the one that I have internally (close to ready, though dependent on the firs from the above) should actually close this.
will close this out by adding a blocking test and docs to
Nice work ... that jq stuff is something :)
/triage accepted


Question: ServiceLoadBalancerFinalizer:   {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, LocalStorageCapacityIsolationFSQuotaMonitoring: {Default: false, PreRelease: featuregate.Alpha}, NonPreemptingPriority:                          {Default: false, PreRelease: featuregate.Alpha}, <del> VolumePVCDataSource:                            {Default: true, PreRelease: featuregate.Beta}, </del> <ins> VolumePVCDataSource:                            {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 </ins> PodOverhead:                                    {Default: true, PreRelease: featuregate.Beta}, IPv6DualStack:                                  {Default: false, PreRelease: featuregate.Alpha}, EndpointSlice:                                  {Default: true, PreRelease: featuregate.Beta},

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } if err := ds.client.StopContainer(podSandboxID, defaultSandboxGracePeriod); err != nil { <del> glog.Errorf("Failed to stop sandbox %q: %v", podSandboxID, err) </del> // Do not return error if the container does not exist if !libdocker.IsContainerNotFoundError(err) { <ins> glog.Errorf("Failed to stop sandbox %q: %v", podSandboxID, err) </ins> errList = append(errList, err) } }

/kind bug What happened: logs when container is gone, should only log when we return an error
/sig node


Question: assert.Equal(t, 1, len(az.nodeNames)) } <ins> func TestUpdateNodeCacheExcludeLoadBalancer(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() az := GetTestCloud(ctrl) zone := fmt.Sprintf("%s-0", az.Location) nodesInZone := sets.NewString("prevNode") az.nodeZones = map[string]sets.String{zone: nodesInZone} az.nodeResourceGroups = map[string]string{"prevNode": "rg"} az.unmanagedNodes = sets.NewString("prevNode") az.excludeLoadBalancerNodes = sets.NewString() az.nodeNames = sets.NewString("prevNode") // excluded node cache shall be updated even if the topology label does not change prevNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ v1.LabelTopologyZone: zone, }, Name: "prevNode", }, } newNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ v1.LabelTopologyZone:         zone, v1.LabelNodeExcludeBalancers: "true", }, Name: "newNode", }, } az.updateNodeCaches(&prevNode, &newNode) assert.Equal(t, 1, len(az.excludeLoadBalancerNodes)) } </ins> func TestGetActiveZones(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish()

Doing a micro upgrade from one level of 1.23 to another level of 1.23, we noticed disruption to PDB based app went from about 5s to about 13s when compared to a 1.22.z to 1.22.z+1 update. I did not expect any disruption. This one is a bit difficult, we have thousands of runs to compare averages and distributions against. It's specifically azure, it's specifically the service load balancer, it's worst on 1.23 to 1.23 upgrades versus 1.22 to 1.23 upgrades. We tracked it down to one of these three commits: When we revert all three, we see improved behavior (our test was in ). We attempted to workaround by setting the exclude-from-lb label, but this filter prevents the change from being observed by the informer's cache: When we attempted to remove the filter entirely (attempt here ), we get from <details</detailsAzure <details</details<details</details<details</details<details</details
quoting from on an internal thread klog.V(2).Infof("reconcileLoadBalancer for service (%s)(%t): lb backendpool - found unwanted node %s, decouple it from the LB", serviceName, wantLb, nodeName)
/sig cloud-provider
ping
/cc
Adding more details to the issue... We analyzed how Azure Load Balancer backends are modified as cluster nodes become or stop being available, which in our tests can happen in two ways: by labeling nodes withby rebooting nodes With kubernetes prior to this PR (, solving ), namely kubernetes 1.22, shipped with openshift 4.9, we notice that: cloud service controller in receives event notifications on nodes and updates accordingly the list of available nodes for load balancers (): a node is excluded from a load balancer backend pool if it has the "exclude" label above, or if its status is not ; in receives the updated list of nodes from the cloud service controller this list differs from the current list of nodes in the backend pool for a given load balancer, it removes spurious nodes and adds any new nodes. particular, node deletion begins here: and takes place in the function.

So far so good, if I label a node that I want to exclude from a load balancer, the following logs appear: Now, if we try to do the same with kubernetes 1.23 (planned to ship with openshift 4.10), we see that the above deletion NEVER takes place: steps 1 and 2 from above are the same: the cloud service controller realizes that a node has been labeled (or rebooted) and removes from the list of nodes it passes to the Azure The PR mentioned above, however, changes the logic of how nodes are removed and the result is that deletion never happens, at least for the two cases mentioned above (a node that gets the "exclude" label, a node that is rebooting). From what I understand in the current code: deletion now happens if this call returns true: this function does NOT make use of the node list received as input by the reconciler. It looks for the node name in , which is updated by , which is in turn triggered by separate node informers in for some reason, these node informers trigger a cache update only if the topology label () on them changes, so the list in is only correct at startup, and then becomes stale. As a consequence, in our tests, as nodes reboot or get labeled, the code always checks for inclusion against an empty list... The update logic with these node informers was not part of the PR mentioned above. Is this something that was done on purpose? In order to overcome this, I tried removing the condition on the topology label and triggering a cache update on every update event. This goes a little further, but deletion is skipped because of : Indeed, if I add debug lines I see that: - I am not familiar with these terms and what they entail. What's interesting, though, is that this code hasn't changed all that much since openshift 4.9, so I debug lines to see what the values for and were back then and it turned out that they were both empty strings! The check for equality was then successful and deletion took place. Now, why were they both empty?

is an input parameter in and is set here, in the reconciler function: In one of my tests, held the same value as (), so in the code below we call : It turns out, returns an empty value in openshift 4.9: This code wasn't modified in the PR or since 1.22, so I do not see how now with 1.23 we finally get a proper value for ... So to recap, The node cache updates take place only when the topology label changes. Was this done on purpose? I don't see it mentioned in the issue linked to the original PReven if we trigger updates to the node cache systematically (disregarding the topology label check), we then hit another problem that prevents the delete operation to take place. This involves some subtleties in the Azure configuration (vmasName vs vmSet, and also Config.PrimaryAvailabilitySetName) and we definitely need someone from the upstream community to take a look at any code changes we might introduce in there...
Here I wrote down the steps needed to reproduce this issue, with the necessary files available in the same repo:
(I just updated my long comment above with more precise info on openshift 4.9)
Thanks, & We'll try to get the right folks looking at this.
Thanks , appreciate all your help and efforts on this one!
I am suspecting there is a conflict between: the original PR in the kubernetes-sigs repo ()the code that eventually got merged in kubernetes 1.23 () The update of the internal cache ( occurring only when the topology label changes might not have been in the intention of the original PR author... In the original pull request, the part of runs every time it gets executed, as I suggested in my comment above: In kubernetes 1.23, this got merged with the (pre-existing) checks on the topology labels that are not included in the kubernetes-sigs repo:
thanks for catching this, I will file a PR to fix it ASAP. By the way, is it possible to switch to out-of-tree cloud provider azure since it has been GA?
/assign
/triage accepted
Let me see if I got this right.

In you say the following: Before was merged, a node was removed from the backend pool of a load balancer if: status was no longer Ready or was labeled with This decision happened in the cloud provider controller: ... and the result of that was passed over to the azure-specific code in that you quote in the PR: Now, with this PR merged, a node gets removed from the backend pool of a load balancer only if: is labeled with or is labeled with ( ) This means that nodes that are no longer Ready still appear in the cloud loadbalancer backend pools, whose contents no longer reflect available nodes. Is this really the desired behaviour? Were there good reasons behind this decision? Could we add a couple of lines in in that also lookup the status of a node? What are your thoughts on this? Thanks a lot!
The fix would be rather simple, something along the lines of What do you guys think? Thanks!


Question: install.Install(audit.Scheme) } <ins> // retryOnError enforces the webhook client to retry requests // on error regardless of its nature. // The default implementation considers a very limited set of // 'retriable' errors, assuming correct use of HTTP codes by // external webhooks. // That may easily lead to dropped audit events. In fact, there is // hardly any error that could be a justified reason NOT to retry // sending audit events if there is even a slight chance that the // receiving service gets back to normal at some point. func retryOnError(err error) bool { if err != nil { return true } return false } </ins> func loadWebhook(configFile string, groupVersion schema.GroupVersion, initialBackoff time.Duration) (*webhook.GenericWebhook, error) { <del> return webhook.NewGenericWebhook(audit.Scheme, audit.Codecs, configFile, </del> <ins> w, err := webhook.NewGenericWebhook(audit.Scheme, audit.Codecs, configFile, </ins> []schema.GroupVersion{groupVersion}, initialBackoff) <ins> w.ShouldRetry = retryOnError return w, err </ins> } type backend struct {

What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery


Question: exit 1 fi <ins> echo "$(date +'%Y-%m-%d %H:%M:%S') Detecting if migration is needed" </ins> if [ "${TARGET_STORAGE}" != "etcd2" -a "${TARGET_STORAGE}" != "etcd3" ]; then echo "Not supported version of storage: ${TARGET_STORAGE}" exit 1

At the very least it should be clear when (timestamp) the migration started and finished.

If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.


Question: S := now.Sub(r.startTime).Seconds() // When a request finishes being served, and the actual service time was S, <del> // the queue’s virtual start time is decremented by G - S. r.queue.virtualStart -= (qs.estimatedServiceTime * float64(r.Seats())) - S </del> <ins> // the queue’s virtual start time is decremented by (G - S)*width. r.queue.virtualStart -= (qs.estimatedServiceTime - S) * float64(r.Seats()) </ins> // request has finished, remove from requests executing r.queue.requestsExecuting--

I reviewed the current code to see what had been done to generalize request width. I found a few inconsistencies in . A really simple one is at line , where is subtracted from the product but should have been subtracted from just (and their difference multiplied by ). With that fixed, a queue's tracks the expected "virtual time" ( in the original paper) when the oldest waiting request will start. However, computes the expected finish "virtual time" of the J'th request by multiplying by but not . There should be a consistent notion of "virtual time" (AKA ), either always including seats or excluding them. Kubernetes version (use ): latest k8s/apiserver sources Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I suspect it would be better if "virtualTime" is not multiplied by request width. I am working on writing.
/triage accepted
I am running into a dilemma. The whole point of "virtual time" --- R(t) in the original paper --- is that it is shared among all the queues. They all get work done at the same rate, dR/dt. But when we generalize so that requests have different widths (and thus work is the sum over requests of ), even restricting the virtual world so that a queue runs only one request at a time does not save us from the fact that different queues get work done at different rates. Thus, even discarding fairness, and executing one request at a time per queue in the virtual world, it looks like what can be shared among queues is not a matter of work done.
But the current implementation approach makes the assumption that all the non-empty queues get work done at the same rate. As long as we are willing to tolerate the lossages due to that fiction, the queues can share one R(t) meter.
I see both sides of this; I still think it is double counting to use to advance virtual time AND to occupy W seats. OTOH, I also think it is right that we should be fair over the work () -- not just duration. One crazy idea is to add the request to multiple queues, and not execute it until it gets to the head of all the queues it is in.

The number of queues it needs to be to would be (# queues) * W / seats, rounded up I guess.
I think "virtual time" is a fuzzy concept that does not come equipped with guidelines about how to think about it correctly. That is why I followed the style of reasoning in the original paper, it defines a specific function and shows why and how that particular function is useful. In the KEP revision I am preparing, I think the problem statement ()the equations defining intended behavior ()the implementation discussion () make sense. It is in the last of those that is introduced, and I think it is clear that this should be a measure that includes request width.
I don't think it's double counting, because those are different things. And yes - we should be fair over WD - this is exactly why I wanted to push virtual time by WD. - I've seen your KEP PR, just I'm under water. My Fridays are real working time, so I will take a look tomorrow.


Question: // and in the process of restarting return api.PodRunning default: <ins> glog.V(5).Infof("pod default case, pending") </ins> return api.PodPending } }

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: return fmt.Errorf("looking up provisioner name for volume %v: %w", vol, err) } if !isCSIMigrationOn(csiNode, inTreeProvisionerName) { <ins> csiNodeName := "" if csiNode != nil { csiNodeName = csiNode.Name } </ins> klog.V(5).InfoS("CSI Migration is not enabled for provisioner", "provisioner", inTreeProvisionerName, <del> "pod", klog.KObj(pod), "csiNode", csiNode.Name) </del> <ins> "pod", klog.KObj(pod), "csiNode", csiNodeName) </ins> return nil } // Do translation for the in-tree volume.

handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.


Question: 'http://metadata/computeMetadata/v1/instance/attributes/kube-env'" 2>/dev/null } <ins> # Read os distro information from /os/release on node. # $1: The name of node # # Assumed vars: #   PROJECT #   ZONE function get-node-os() { gcloud compute ssh "$1"  --project "${PROJECT}"  --zone "${ZONE}"  --command  "cat /etc/os-release | grep "^ID=.*" | cut -c 4-" } </ins> # Assumed vars: #   KUBE_VERSION #   NODE_SCOPES

During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm.

If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.


Question: return &TestDuplicatePlugin{}, nil } <ins> // TestPermitPlugin only implements PermitPlugin interface. type TestPermitPlugin struct { PreFilterCalled int } func (pp *TestPermitPlugin) Name() string { return permitPlugin } func (pp *TestPermitPlugin) Permit(ctx context.Context, state *CycleState, p *v1.Pod, nodeName string) (*Status, time.Duration) { return NewStatus(Wait, ""), time.Duration(10 * time.Second) } </ins> var registry Registry = func() Registry { r := make(Registry) r.Register(scoreWithNormalizePlugin1, newScoreWithNormalizePlugin1)

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: csics := f.CSIClientSet var err error <del> m.driver = drivers.InitMockCSIDriver(tp.registerDriver, tp.attachable, tp.podInfoVersion, tp.attachLimit) </del> <ins> m.driver = drivers.InitMockCSIDriver(tp.registerDriver, !tp.disableAttach, tp.podInfoVersion, tp.attachLimit) </ins> config, testCleanup := m.driver.PrepareTest(f) m.testCleanups = append(m.testCleanups, testCleanup) m.config = config

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear


Question: } for _, test := range tests { t.Run(test.name, func(t *testing.T) { <ins> assignDefaultStartTime(test.pods) </ins> nodes := []*v1.Node{} for _, n := range test.nodes { nodes = append(nodes, makeNode(n, priorityutil.DefaultMilliCPURequest*5, priorityutil.DefaultMemoryRequest*5))

What would you like to be : TestSelectNodesForPreemption should assign pod.Status.StartTime for test Pod Why is this needed: We can see the following in the output of TestSelectNodesForPreemption: This is due to missing assignment to Status.StartTime
/sig scheduling
/assign


Question: // 172.17.0.1 dev docker0  scope host  src 172.17.0.1 // 192.168.122.1 dev virbr0  scope host  src 192.168.122.1 // Then filter out dev==kube-ipvs0, and cut the unique src IP fields, <del> // Node IP set: [100.106.89.164, 127.0.0.1, 172.17.0.1, 192.168.122.1] </del> <ins> // Node IP set: [100.106.89.164, 172.17.0.1, 192.168.122.1] // Note that loopback addresses are excluded. </ins> func (r *realIPGetter) NodeIPs() (ips []net.IP, err error) { // Pass in empty filter device name for list all LOCAL type addresses. nodeAddress, err := r.nl.GetLocalAddresses("", DefaultDummyDevice)

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: cs = f.ClientSet }) <del> ginkgo.It("should set default value on new IngressClass", func() { </del> <ins> ginkgo.It("should set default value on new IngressClass [Serial]", func() { </ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", true, f.UniqueName) framework.ExpectNoError(err) defer deleteIngressClass(cs, ingressClass1.Name)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in


Question: isCreated bool } <del> func parseKV(kv *mvccpb.KeyValue, prevVal []byte) *event { </del> <ins> // parseKV converts a KeyValue retrieved from an initial sync() listing to a synthetic isCreated event. func parseKV(kv *mvccpb.KeyValue) *event { </ins> return &event{ key:       string(kv.Key), value:     kv.Value, <del> prevValue: prevVal, </del> <ins> prevValue: nil, </ins> rev:       kv.ModRevision, isDeleted: false, <del> isCreated: kv.ModRevision == kv.CreateRevision, </del> <ins> isCreated: true, </ins> } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key.

What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring.

The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.


Question: --kubelet-port=${KUBELET_PORT}  --runtime-config=api/v1  --cert-dir="${TMPDIR:-/tmp/}"  <ins> --runtime_config="extensions/v1beta1=true"  </ins> --service-cluster-ip-range="10.0.0.0/24" 1>&2 & APISERVER_PID=$!

Now that we've moved the experimental api to /apis/experimental... it's inaccessible from by default. We should fix that. who stepped on this earlier
security questions for you: this prefix was apparently in response to a security review quite some time ago. I don't understand how this adds much security. Can we just remove it? If not, is it acceptable to change the default from "/api/" to "/api", which would match both api locations? Otherwise we'll have to take a comma-separated list or something...
Ref
Oh, so this is not related to the security requests! And the original use case was just to re-root the URL path so it didn't interfere with static content. Sorry, I got confused because the last time I monkeyed with that, it was in conjunction with the allow/deny list that actually was the security request. In that case, let's see if we can change the default to "/", if that will break anything.
That change got introduced as part of to protect against XSRF in kubectl proxy. But I think the more important part of that change was that it disallows POST/PUT/PATCH by default. So as long as there is no state changing action outside of /api/ that can be triggered with a GET it should be safe to remove.


Question: tearDownUserspaceIptables(ipt) return &Proxier{ <del> serviceMap: make(map[proxy.ServicePortName]*serviceInfo), syncPeriod: syncPeriod, iptables:   ipt, </del> <ins> serviceMap:    make(map[proxy.ServicePortName]*serviceInfo), syncPeriod:    syncPeriod, iptables:      ipt, MasqueradeAll: MasqueradeAll, </ins> }, nil }

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote: