Question: objectMeta.SetDeletionGracePeriodSeconds(nil) strategy.PrepareForCreate(ctx, obj) FillObjectMetaSystemFields(objectMeta) if len(objectMeta.GetGenerateName()) > 0 && len(objectMeta.GetName()) == 0 { objectMeta.SetName(strategy.GenerateName(objectMeta.GetGenerateName())) }

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: serviceNamespace: apiService.Spec.Service.Namespace, serviceAvailable: apiregistrationapi.IsAPIServiceConditionTrue(apiService, apiregistrationapi.Available), } <ins> if r.proxyTransport != nil && r.proxyTransport.DialContext != nil { newInfo.restConfig.Dial = r.proxyTransport.DialContext } </ins> newInfo.proxyRoundTripper, newInfo.transportBuildingError = restclient.TransportFor(newInfo.restConfig) <del> if newInfo.transportBuildingError == nil && r.proxyTransport != nil && r.proxyTransport.DialContext != nil { switch transport := newInfo.proxyRoundTripper.(type) { case *http.Transport: transport.DialContext = r.proxyTransport.DialContext default: newInfo.transportBuildingError = fmt.Errorf("unable to set dialer for %s/%s as rest transport is of type %T", apiService.Spec.Service.Namespace, apiService.Spec.Service.Name, newInfo.proxyRoundTripper) glog.Warning(newInfo.transportBuildingError.Error()) } </del> <ins> if newInfo.transportBuildingError != nil { glog.Warning(newInfo.transportBuildingError.Error()) </ins> } r.handlingInfo.Store(newInfo) }

/kind bug What happened: kube-aggregrator returns 500 status code with message: unable to set dialer for wardle/api as rest transport is of type *transport.debuggingRoundTripper What you expected to happen: 500 status code should not returned How to reproduce it (as minimally and precisely as possible): kube-apiserver with flag: sample-apiserver as said. command and get the 500 status code: Environment: community master
/assign
/sig api-machinery
/cc


Question: "k8s.io/client-go/util/csaupgrade" "k8s.io/component-base/version" "k8s.io/klog/v2" <del> "k8s.io/kubectl/pkg/cmd/delete" </del> <ins> cmddelete "k8s.io/kubectl/pkg/cmd/delete" </ins> cmdutil "k8s.io/kubectl/pkg/cmd/util" "k8s.io/kubectl/pkg/scheme" "k8s.io/kubectl/pkg/util"

As per , I'm trying to remove CPU limits. But the behaviour is different depending whether the ResourceQuota exists or not: I would expect to be idempotent, and for its behaviour to result in the same end configuration regardless of the initial configuration. a new namespace named this command: that the ResourceQuota is created and is set to it again that is removed each subsequent time it's run, as long as resourcequota/compute-resource exists resourcequota/compute-resource again that the ResourceQuota is created and is set to again, and so forth the exact same behaviour if is changed to No response <details</details<detailsOpenShift 4.13.12 </details<details</details<details</details<details</details<details</details
/sig node
Tried to replicate the issue and observed the same behaviour.
/remove-sig node /sig api-machinery /cc
What's the goal of applying here? What are you trying to achieve? I agree that this is not the right behavior though.
I'm trying to remove a previously set property. I cannot remove the ResourceQuota altogether because it contains other properties (which I left out of my example) that I do not wish to remove.
But 0 is the same as unset, isn't it?
No, if I set it to 0 then any deployment with set to higher than 0 will fail to create any pods because it will exceed the quota.
/triage accepted /cc
/assign
Hello, I'm having a similar issue. Here's my use case: We use Flux to deploy things on our cluster - in particular, we use the component and set chart values in . Some of the charts we deploy have default values we want to remove, so we want to set a value to Editing an HelmRelease with this kind of ajustments drops the null keys, so default values end up being setup again and generated manifests are invalid :( Let me know if you need anything else to keep this issue moving!
I think the correct behavior is to set the struct to the default value when or , because in the standard library does this as well, and I think we expect the result to be as similar as possible to . So our real issue here is that we didn't set the to its default value when patch.
Based on the above, I think this question is not related to the current issue.

Tracing it back to , maybe the current result is what we want. The explicit null values means that we want to remove it in patch. As mentioned , our behavior in create is also as expected. I'm not quite sure if we actually want to fix it, and how /cc I hope to get your advice as you are the implementer of
I think client-side on create should do what on update will accomplish, namely, explicit nulls in the manifest are treated as the removal of the attributes they are values of. Proof of concept opened at , I'd be happy for someone to pick that up, add test coverage, and work through any failing tests that change uncovers
Thank you, but I still have a question. If we solve the difference between create and patch behaviors in ApiServer on the client side, will third-party callers still have the same problem? Or should we not cover this scenario?
I would consider changing behavior of null in a create request server-side to be a compatibility break. values are defined for client-side apply to mean "remove the key associated with this value". I think fixing the client-side apply creation case to have the same semantic is the most narrowly scoped fix we can make while preserving compatible server-side behavior generally.
Okay, I got it, I'll follow up with


Question: spec: containers: - name: pause-win <del> image: gcr.io/gke-release/pause-win:1.2.0 </del> <ins> image: gcr.io/gke-release/pause-win:1.5.1 </ins> nodeSelector: kubernetes.io/os: windows tolerations:

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add Support for Windows Server version 2004 and 20H2. Why is this needed: Windows Server version 1909 is going end of life.
/sig windows /triage accepted


Question: }) } <ins> func waitForConfigMapInNamespace(c clientset.Interface, ns, name string, timeout time.Duration) error { fieldSelector := fields.OneTermEqualSelector("metadata.name", name).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (object runtime.Object, e error) { options.FieldSelector = fieldSelector return c.CoreV1().ConfigMaps(ns).List(context.TODO(), options) }, WatchFunc: func(options metav1.ListOptions) (i watch.Interface, e error) { options.FieldSelector = fieldSelector return c.CoreV1().ConfigMaps(ns).Watch(context.TODO(), options) }, } ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) defer cancel() _, err := watchtools.UntilWithSync(ctx, lw, &v1.ConfigMap{}, nil, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: return false, apierrors.NewNotFound(schema.GroupResource{Resource: "configmaps"}, name) case watch.Added, watch.Modified: return true, nil } return false, nil }) return err } </ins> func waitForServiceAccountInNamespace(c clientset.Interface, ns, serviceAccountName string, timeout time.Duration) error { fieldSelector := fields.OneTermEqualSelector("metadata.name", serviceAccountName).String() lw := &cache.ListWatch{

Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /cc
/hold I have a subtly different fix in :)
Oh wait, this is an issue. Yay, thanks :). Fixed in


Question: fs.BoolVar(&s.EnableProfiling, "profiling", true, "Enable profiling via web interface host:port/debug/pprof/.") fs.BoolVar(&s.EnableContentionProfiling, "contention-profiling", false, "Enable lock contention profiling, if profiling is enabled.") fs.StringVar(&s.ClusterCIDR, "cluster-cidr", s.ClusterCIDR, "CIDR Range for Pods in cluster.") <ins> fs.StringVar(&s.ClusterName, "cluster-name", s.ClusterName, "The instance prefix for the cluster.") </ins> fs.BoolVar(&s.AllocateNodeCIDRs, "allocate-node-cidrs", false, "Should CIDRs for Pods be allocated and set on the cloud provider.") fs.StringVar(&s.Master, "master", s.Master, "The address of the Kubernetes API server (overrides any value in kubeconfig).") fs.StringVar(&s.Kubeconfig, "kubeconfig", s.Kubeconfig, "Path to kubeconfig file with authorization and master location information.")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is used by servicecontroller and routecontroller, for controller-manager, we have an option to set it, but for cloud-controller-manager, it will always be of default value 'kubernetes'. Here are some examples where it was in use: aws: azure: gce: openstack What you expected to happen: Having an option to set the value. How to reproduce it (as minimally and precisely as possible):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cluster-lifecycle /area cloudprovider
oops looks like a PR ready already :)


Question: --system-cgroups=/system  --cgroup-root=/  --network-plugin=$NETWORK_PLUGIN  <del> --network-plugin-dir=$NETWORK_PLUGIN_PATH  </del> <ins> --cni-conf-dir=$CNI_CONF_DIR  --cni-bin-dir=$CNI_BIN_DIR  </ins> --v=$log_level  --logtostderr

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: return types.NodeName(aws.StringValue(i.PrivateDnsName)) } <ins> var aliveFilter = []string{ ec2.InstanceStateNamePending, ec2.InstanceStateNameRunning, ec2.InstanceStateNameShuttingDown, ec2.InstanceStateNameStopping, ec2.InstanceStateNameStopped, } </ins> // Returns the instance with the specified node name // Returns nil if it does not exist func (c *Cloud) findInstanceByNodeName(nodeName types.NodeName) (*ec2.Instance, error) { privateDNSName := mapNodeNameToPrivateDNSName(nodeName) filters := []*ec2.Filter{ newEc2Filter("private-dns-name", privateDNSName), <del> newEc2Filter("instance-state-name", "running"), </del> <ins> // exclude instances in "terminated" state newEc2Filter("instance-state-name", aliveFilter...), </ins> } instances, err := c.describeInstances(filters)

If a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster. Looking at the various cloud providers, some of them filter nodes based on the state whereas others do not. The node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed. Is there a reason why the cloud providers only return instances/vms that are "ready" instead of returning any instance/vm that matches the requested name? It seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.
If that's really the case it's unfortunate - we probably can fix logic to list everything (I guess there's a flag that you can pass to CP to give you everything, not only working machines). I have no idea who deals with that ? OTOH I think the only problem with this would be if the machine would flap, as this would be annoying and cluster autoscaler might get crazy ( If it's really unready/unavailable/whatever then it's fair to assume that nothing's working on it (if StatefulSet can live with that assumption
do you think the NodeController should/should not delete a Node from storage in this situation: Node is configured to use a cloud provider, VM is powered off but not deleted? I would argue for "should not delete".
A trivial answer is that PrivateDNSNames are not unique on AWS. A running instance and a stopped instance could have the same IP, hence same PrivateDNSName. Rob / Marek: Can you clarify with a scenario where something unexpected happens: what that is and what you expect to happen?
can we work around that somehow for ec2?
I don't know what we are working around! The concern is AWS instances that are Stopped but not Terminated? Separately, we've been working since before 1.0 to get us off PrivateDNSName on AWS though...
the issue is that, depending on cloud provider, the NodeController may delete a Node from etcd if the VM is not Running.

If you power off a VM, wait longer than the NodeController's eviction threshold (5 minutes by default), and then power the VM back on later, the NodeController will have deleted the Node from etcd in the interim. Some cloud providers in the kube code base expect an instance to be Running, and if it's not, they don't include that instance in the list of instances, or a "get instance" call returns "not found" (I'm not sure which, has the full details), and so the NodeController doesn't see the powered off VM and deletes the Node.
That is the current behaviour, but you've skipped the bit where something goes wrong :-) Even if we fix this problem though do we close this issue? I would vote to close this issue now on the basis that there's no objective measure for when we can mark this issue resolved, and that we start again with a problem statement like the one you're describing, which feels actionable.
the intent of this issue was what I described. It's possible that it wasn't clear in the initial description.
At least on AWS: The motivation for the filtering is that Terminated instances still have the old PrivateDNSName listed (IIRC), and so the names aren't unique unless you exclude Terminated. But we don't have to exclude Stopped instances (power-off), and the nodes will just get marked NotReady. I don't think that will cause problems, but it may be that the correct place to fix is actually somewhere else - e.g. on kubelet rejoin. What goes wrong when the node tries to rejoin?
At least 1 thing: if you have manually labels to the node post-registration, those labels are gone when it rejoins.
That sounds like it would be correct for some classes of labels.
- small clarification: 5 min threshold has nothing to do here. It only says that if Node is unresponsive for 5 or more minutes we'll start evicting Pods from that Node (modulo rate limiting). Deleting non-existing Nodes is a separate thing - when Node skips few heartbeats (currently 4) NC checks if cloud provider still returns the instance corresponding to that Node, and if not then delete the Node instance. IIUC what wrote the current behavior is valid for AWS, as Node won't come back (it's DNS name might be used by something else).

This isn't the case for only aws. Looking at the code it appears at least openstack, aws, vsphere, ovirt, rackspace cloud providers also filter on available node. I wasn't able to verify whether azure or others were doing the filtering. If they weren't, then at best we have inconsistency with different cloud providers. IMO we shouldn't be deleting a node from kubernetes if it is just stopped or some state other than available. If the instance exists in the cloud provider then imo it should exist in kubernetes, but possibly marked unscheduable if needed.
ouch, that's even worse! Isn't that only 40-50 seconds? and I chatted yesterday and I think there are a couple of possible ways to look at this: 'm used to doing things the "old way" where I treat my servers as pets, and I want to be the one who decides when to retire/delete a server. 'm willing to treat servers as cattle, and if I ever need to take a server down so I can upgrade the kernel/packages/etc, or if a server goes offline for whatever reason, I'll just throw it away and replace it with a newer one. I'd be willing to bet that most IT organizations are still operating under 1). Justin recommended that we need to do a better job at education, and while I agree with that, there is still going to be a transitional period where sysadmins want to be making these kinds of decisions, and they're not yet willing to let the cluster automatically delete nodes. In summary, I think at the very least we need to: this behavior, if it's not already documented an option to the node controller to disable this behavior (and possibly disable it by default) more on evangelizing how we envision server management to happen in a Kubernetes world
Yes, I believe it's 40s. I'm curious why we don't want to just change the implementation of cloud provider, so that we'll use 'full list' (i.e. one including not running Nodes, so only delete ones that are really not present) in NC?
That also works for me, assuming it doesn't cause any issues.
It's strictly better than disabling it altogether, right?

The drawback would be that if you create a VM which will get the same name (I guess it's a DNS name), than the suspended/whatever one, then it'll be treated by k8s as the old one, which may cause a issues when running StatefulSets (especially if old machine didn't release PV or sth, as IIUC k8s will believe that it's attached to the new machine But if you're willing to do that, then I don't see anything that will prevent you from doing so, assuming it will be flag gated.
I think stateful sets are unaffected by this unless both new and old machines are booted at the same time AND either kubelet fails to terminate the pods. I can't think of how that would happen. Not being able to unmount a volume could be caused by other admin actions and is something we can warn about.
For a change of this significance, we'll want an e2e showing the problem, and then a PR that lets the e2e pass.
I'm not sure an e2e is the best way to show this. An e2e would incur costs to run on a cloud provider so likely wouldn't be part of any standard e2e run. I think a unit test could be done though.
The PR I pushed only changes AWS. w/o this change the node is removed from kubernetes shortly after it is no longer running. With this PR the node transitions to unschedulable and details show it hasn't reported its health check.
PTAL and let me know if you see issues with this approach
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale


Question: secret := GetObjectFromChan(c).(*api_v1.Secret) return secret } <ins> // Wait till the store is updated with latest secret. func WaitForSecretStoreUpdate(store util.FederatedReadOnlyStore, clusterName, key string, desiredSecret *api_v1.Secret, timeout time.Duration) error { retryInterval := 100 * time.Millisecond err := wait.PollImmediate(retryInterval, timeout, func() (bool, error) { obj, found, err := store.GetByKey(clusterName, key) if !found || err != nil { return false, err } equal := secretsEqual(*obj.(*api_v1.Secret), *desiredSecret) return equal, err }) return err } </ins>

https://k8s- Failed: TestSecretController {secret} Previous issues for this test:
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
This flake seems to have started after merged (just looking at the merge time line). Can you please take a look?
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
https://k8s- Failed: TestSecretController {secret}
Sorry just saw this, looking at it now
Sent for a fix
nice thanks


Question: }, status: "Unknown,SchedulingDisabled", }, <ins> { node: api.Node{ ObjectMeta: metav1.ObjectMeta{ Name:   "foo12", Labels: map[string]string{"kubeadm.alpha.kubernetes.io/role": "node"}, }, Status: api.NodeStatus{Conditions: []api.NodeCondition{{Type: api.NodeReady, Status: api.ConditionTrue}}}, }, status: "Ready,node", }, </ins> } for _, test := range table {

used to show masters, but as of it no longer does. This is not in accordance with our deprecation policy I believe. cc
That commit/PR is not relevant as there are no supported users/deployment methods that are using label. kubeadm v1.4 or v1.5 was the last. What you're really looking for is who/what removed printing things based on the label...
No blame on you The regression is that we have stopped printing masters in . There may be more to it that a simple revert of , but we should probably start there.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed in recent versions


Question: fs.DurationVar(&c.HTTPCheckFrequency.Duration, "http-check-frequency", c.HTTPCheckFrequency.Duration, "Duration between checking http for new data") fs.StringVar(&c.StaticPodURL, "manifest-url", c.StaticPodURL, "URL for accessing additional Pod specifications to run") fs.Var(cliflag.NewColonSeparatedMultimapStringString(&c.StaticPodURLHeader), "manifest-url-header", "Comma-separated list of HTTP headers to use when accessing the url provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'") <del> fs.Var(utilflag.IPVar{Val: &c.Address}, "address", "The IP address for the Kubelet to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)") </del> <ins> fs.Var(utilflag.IPVar{Val: &c.Address}, "address", "The IP address for the Kubelet to serve on (set to '0.0.0.0' or '::' for listening in all interfaces and IP families)") </ins> fs.Int32Var(&c.Port, "port", c.Port, "The port for the Kubelet to serve on.") fs.Int32Var(&c.ReadOnlyPort, "read-only-port", c.ReadOnlyPort, "The read-only port for the Kubelet to serve on with no authentication/authorization (set to 0 to disable)")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are flag descriptions where is stated that the unspecified address behaves differently depending on the IP family: However, that's not the behavior implemented in and , clarified in the golang documentation What you expected to happen: Fix the comment to state the reality, i.e How to reproduce it (as minimally and precisely as possible): From the root of the kubernetes repo Anything else we need to know?: xref Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind cleanup /help wanted /good-first-issue /priority important-longterm /cc
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign /remove-help
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
For the flag in kube-proxy, is used if the address is not specified and can't be resolved by hostname, so I suggest to remove this comment: 0.0.0.0::` for all IPv6 interfaces'
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I am very much in favor of this. Being correct and precise in these flags is valuable. To convince myself you were right, I wrote this program: And then I ran it for various inputs: In these flags and docs, we should be clear that if you are specifying a v6 addr and port you have to use syntax. Or maybe we should just audit and fix ALL such places to support and document that "nothing" is how you specify all interfaces. E.g. --bind-address=":80" is more obvious than either "0.0.0.0:80" or "[::]:80".

Even if those syntaxes work, let's just make "" the norm?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil, err } if e.Decorator != nil { <del> return newDecoratedWatcher(w, e.Decorator), nil </del> <ins> return newDecoratedWatcher(ctx, w, e.Decorator), nil </ins> } return w, nil }

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: } } <ins> // TestNoOpApplyWithEmptyMap func TestNoOpApplyWithEmptyMap(t *testing.T) { client, closeFn := setup(t) defer closeFn() deploymentName := "no-op" deploymentsResource := "deployments" deploymentBytes := []byte(`{ "apiVersion": "apps/v1", "kind": "Deployment", "metadata": { "name": "` + deploymentName + `", "labels": { "app": "nginx" } }, "spec": { "replicas": 1, "selector": { "matchLabels": { "app": "nginx" } }, "template": { "metadata": { "annotations": {}, "labels": { "app": "nginx" } }, "spec": { "containers": [{ "name": "nginx", "image": "nginx:1.14.2", "ports": [{ "containerPort": 80 }] }] } } } }`) _, err := client.AppsV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource(deploymentsResource). Name(deploymentName). Body(deploymentBytes). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } // This sleep is necessary to consistently produce different timestamps because the time field in managedFields has // 1 second granularity and if both apply requests happen during the same second, this test would flake. time.Sleep(1 * time.Second) createdObject, err := client.AppsV1().RESTClient().Get().Namespace("default").Resource(deploymentsResource).Name(deploymentName).Do(context.TODO()).Get() if err != nil { t.Fatalf("Failed to retrieve created object: %v", err) } createdAccessor, err := meta.Accessor(createdObject) if err != nil { t.Fatalf("Failed to get meta accessor for created object: %v", err) } createdBytes, err := json.MarshalIndent(createdObject, "t", "t") if err != nil { t.Fatalf("Failed to marshal created object: %v", err) } // Test that we can apply the same object and don't change the RV _, err = client.AppsV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource(deploymentsResource). Name(deploymentName). Body(deploymentBytes). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } updatedObject, err := client.AppsV1().RESTClient().Get().Namespace("default").Resource(deploymentsResource).Name(deploymentName).Do(context.TODO()).Get() if err != nil { t.Fatalf("Failed to retrieve updated object: %v", err) } updatedAccessor, err := meta.Accessor(updatedObject) if err != nil { t.Fatalf("Failed to get meta accessor for updated object: %v", err) } updatedBytes, err := json.MarshalIndent(updatedObject, "t", "t") if err != nil { t.Fatalf("Failed to marshal updated object: %v", err) } if createdAccessor.GetResourceVersion() != updatedAccessor.GetResourceVersion() { t.Fatalf("Expected same resource version to be %v but got: %vnold object:n%vnnew object:n%v", createdAccessor.GetResourceVersion(), updatedAccessor.GetResourceVersion(), string(createdBytes), string(updatedBytes), ) } } // TestApplyEmptyMarkerStructDifferentFromNil func TestApplyEmptyMarkerStructDifferentFromNil(t *testing.T) { client, closeFn := setup(t) defer closeFn() podName := "pod-with-empty-dir" podsResource := "pods" podBytesWithEmptyDir := []byte(`{ "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "` + podName + `" }, "spec": { "containers": [{ "name":  "test-container-a", "image": "test-image-one", "volumeMounts": [{ "mountPath": "/cache", "name": "cache-volume" }], }], "volumes": [{ "name": "cache-volume", "emptyDir": {} }] } }`) _, err := client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource(podsResource). Name(podName). Body(podBytesWithEmptyDir). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } // This sleep is necessary to consistently produce different timestamps because the time field in managedFields has // 1 second granularity and if both apply requests happen during the same second, this test would flake. time.Sleep(1 * time.Second) createdObject, err := client.CoreV1().RESTClient().Get().Namespace("default").Resource(podsResource).Name(podName).Do(context.TODO()).Get() if err != nil { t.Fatalf("Failed to retrieve created object: %v", err) } createdAccessor, err := meta.Accessor(createdObject) if err != nil { t.Fatalf("Failed to get meta accessor for created object: %v", err) } createdBytes, err := json.MarshalIndent(createdObject, "t", "t") if err != nil { t.Fatalf("Failed to marshal created object: %v", err) } podBytesNoEmptyDir := []byte(`{ "apiVersion": "v1", "kind": "Pod", "metadata": { "name": "` + podName + `" }, "spec": { "containers": [{ "name":  "test-container-a", "image": "test-image-one", "volumeMounts": [{ "mountPath": "/cache", "name": "cache-volume" }], }], "volumes": [{ "name": "cache-volume" }] } }`) // Test that an apply with no emptyDir is recognized as distinct from an empty marker struct emptyDir. _, err = client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource(podsResource). Name(podName). Body(podBytesNoEmptyDir). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } updatedObject, err := client.CoreV1().RESTClient().Get().Namespace("default").Resource(podsResource).Name(podName).Do(context.TODO()).Get() if err != nil { t.Fatalf("Failed to retrieve updated object: %v", err) } updatedAccessor, err := meta.Accessor(updatedObject) if err != nil { t.Fatalf("Failed to get meta accessor for updated object: %v", err) } updatedBytes, err := json.MarshalIndent(updatedObject, "t", "t") if err != nil { t.Fatalf("Failed to marshal updated object: %v", err) } if createdAccessor.GetResourceVersion() == updatedAccessor.GetResourceVersion() { t.Fatalf("Expected different resource version to be %v but got: %vnold object:n%vnnew object:n%v", createdAccessor.GetResourceVersion(), updatedAccessor.GetResourceVersion(), string(createdBytes), string(updatedBytes), ) } } </ins> func getRV(obj runtime.Object) (string, error) { acc, err := meta.Accessor(obj) if err != nil {

Every time I server-side apply a with , the updates. Nothing about my manifest changes between applies; if I map to the empty map, the same manifest updates the . With client-side apply, the stays the same as long as my manifest remains constant, even if . I read in that Kubernetes should treat absent keys, empty values (in this case, an empty map), and values the same. I observe that leaving the key out and behave as expected, not modifying the and leaving the same, but empty map behaves as described above. If we cannot change this behavior, I'd at least like to understand the rationale behind it. I expected my object to stay the same between identical server-side applies, where the manifest file content was exactly the same, regardless of the value of . A different every apply indicates that the object was event type . I expect no events to fire on applying the same manifest. contains In actuality, any is fine. . Every successive repeat of steps 3-4 should show different values. Example of with : This is especially a problem when I server-side apply with code in my operator. My operator is built on and watches changes to the child objects it deploys. If I create a with , the operator constantly reconciles because every server-side apply of the manifest registers as a type event on the object, which triggers the watching operator to reconcile. N/A <details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
Potentially similar case as


Question: } <del> func (s *SecureServingOptions) AddDeprecatedFlags(fs *pflag.FlagSet) { fs.IPVar(&s.BindAddress, "public-address-override", s.BindAddress, "DEPRECATED: see --bind-address instead.") fs.MarkDeprecated("public-address-override", "see --bind-address instead.") } </del> // ApplyTo fills up serving information in the server configuration. func (s *SecureServingOptions) ApplyTo(c *server.Config) error { if s == nil {

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: } type MetadataOpts struct { <del> SearchOrder string `gcfg:"search-order"` </del> <ins> SearchOrder    string     `gcfg:"search-order"` RequestTimeout MyDuration `gcfg:"request-timeout"` </ins> } // OpenStack is an implementation of cloud provider Interface for OpenStack.

What happened: We use OpenStack as our cloud provider. We meet an issue that some controllers stuck because all workers routine stuck at http call to OpenStack services because of rabbitmq issues. Stuck routine call stack: Checked the openstack cloud provider that there is no HttpClient.Timeout set which means no timeout during http call. I think we should set at least the default http timeout value. Environment: Kubernetes version (use ): v1.6.3 Cloud provider or hardware configuration**: OpenStack
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /kind feature PTAL PR .


Question: record_command run_kubectl_explain_tests fi <ins> ############################## # CRD Deletion / Re-creation # ############################## if kube::test::if_supports_resource "${namespaces}" ; then record_command run_crd_deletion_recreation_tests fi </ins> ########### # Swagger #

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Changing the scope field of a CRD ist not possible since it's immutable. But even after deleting and re-creating the CRD with changed scope it doesn't let me create resources of the CRD's kind. Re-creating CRD with changed scope should just work. Here is a full example that in the end leads to the error when trying to create a resource after re-creating the CRD. Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: vSphere 7.0 OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1160.42.2.el7.x86_64
/sig api-machinery
/assign /triage accepted
This appears to be working as intended. The example custom resource you use: It Cluster scoped because it has no namespace. But once the CRD is namespace scoped, you should not be able to create cluster scoped custom resources of that type. If instead you add a namespace to Example, it should be created, e.g.:
if the above seems to have answered your question - please feel free to close the issue :) Thank you!
kubectl will automatically use the namespace so this is definitely not the issue here. I can reproduce this issue with my above script in a fresh (v1.22.2) cluster. You can add the namespace in the last manifest, it makes no difference. From my understanding this really seems to be a bug.
Yes, that's true. Good catch. Okay, we need to root cause this.
I tried using curl to create the custom resource directly against the API and got the same result: So the problem seems to be in the API server (not in kubectl).
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return attached, nil } <del> dnsNameSlice := []string{} </del> <ins> nodeNames := []string{} </ins> for nodeName, diskNames := range nodeDisks { for _, diskName := range diskNames { setNodeDisk(attached, diskName, nodeName, false) } <del> dnsNameSlice = append(dnsNameSlice, mapNodeNameToPrivateDNSName(nodeName)) </del> <ins> nodeNames = append(nodeNames, mapNodeNameToPrivateDNSName(nodeName)) </ins> } <del> awsInstances, err := c.getInstancesByNodeNames(dnsNameSlice) </del> <ins> // Note that we get instances regardless of state. // This means there might be multiple nodes with the same node names. awsInstances, err := c.getInstancesByNodeNames(nodeNames) </ins> if err != nil { // When there is an error fetching instance information // it is safer to return nil and let volume information not be touched.

In we stopped filtering nodes by their running state. But this means that the NodeName is no longer a unique key. We need to rework the volume status checker to deal with this.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Was discussing this with offline. Why can't we extend the filter to exclude terminated ones? I understand that AWS API doesn't provide a documented way of doing negative filtering, but can we not simply list all the known states we care about? such as - , , etc.
We can do that too :-)


Question: exit 1 fi fi <del> params+=" --iptables-sync-period=1m --iptables-min-sync-period=10s --ipvs-sync-period=1m --ipvs-min-sync-period=10s" </del> <ins> params+=" --iptables-sync-period=1m --iptables-min-sync-period=1s --ipvs-sync-period=1m --ipvs-min-sync-period=1s" </ins> if [[ -n "${KUBEPROXY_TEST_ARGS:-}" ]]; then params+=" ${KUBEPROXY_TEST_ARGS}" fi

Analyzing the flake in every change will be queued until the minimum period has expired, The test works as follows: create a NodePort Service with one backend Pod check NodePort Service works make the backend Pod unready check that the NodePort Service doesn't work (kube-proxy should not forward traffic to the unready Pod) The test races because it tries to get to the NodePort before the kube-proxy programs the data-plane and remove the iptables-rules. Sometimes the test is faster than kube-proxy and the pod replies, the request succeeds , causing the test to fail At Nov 29 15:40:06.910: FAIL: expected error when trying to connect to NodePort address Expected an error to have occurred. the NodePort still works Kubelet The pod becomes unready at and the status is updated Controller-manager The endpoint slice controller process the pod update on the EndpointSlices at Kube/proxy here is the thing, kube-proxy queues the iptables update for 4 seconds and race with the test Kube-proxy 2 different proxies queue the call too, but at different times This explains flakes on this tests, since the pod is unready but still answer traffic ... I need to find since when and why are we using 10s as EDIT It seems that it was 10s for a long time in CI Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-soon
/sig testing The question is , why it started to be more evident after 1.25? I want to think that our testing framework is faster after moving to ginkgo v2, does it make sense?
this affects scalability tests IIUIC
I don't think that Ginkgo is involved much once this particular test runs: There are few calls, but that's it. What may have changed is the scheduling of concurrent tests, which may have an effect on the timing.

I was wondering if the code within a spec passes some kind of preprocessing that influences the time to execute each line of the test, I'm trying to make sense of this
I'll try to stare at this some more this morning. But one question - these are running in parallel with other tests, correct? Is it possible that a different container from a different test is coming up and getting assigned the same port? How do you all divvy up ports to ensure that kind of cross-test pollution never happens?
no no , sorry, the test is correct , sorry I'm misdirecting the issue, the thing I wanted to clarify is if there is some overhead when you have a test where the time to execute the or the time between and is affected
The handling of changed. This affects any log output produced by this test. I can't say whether it got faster, slower, or even is noticeable at all.
No - with the caveat that it's always possible that something weird/random is happening - nothing's changed that would cause a systematic slow down or speed up in that way. The By's aren't doing anything slow/fast - and you can comment them out for now if you want to rule it out but, truly, i doubt that's the source of any change.
just wanted to be sure, thanks for confirming
/remove-sig testing
Not sure why this would have started flaking more in 1.25, but anyway, there's a const for "the maximum time a kube-proxy daemon on a node is allowed to not notice a Service update, such as type=NodePort". It is currently 5 minutes, which the comment notes is way too large. But anyway, any test that depends on waiting for kube-proxy to process a Service/Endpoint change should poll for / waiting for it to happen.

yeah, but I don't want to carry with those hacks, we are in much better shape now regarding stability and I want to increase the bar so we detect subtles bugs ... we don't want test that just pass no matter what, we want test to give us confidence things work as expected ... if I have to create a service and wait for 5 minutes to start working we are doing something wrong
oh, I think that I'm doing a mistake, and you are spot on as always the test waits for the Service to not answer, and the connections to the ClusterIP fail, is the NodePort only the problem ... can we have some problem with the 1.25 improvements related to NodePorts only? nevermind, the ClusterIP check depends no the local kube-proxy, the NodePort depends on the kube-proxy of the respective Node, in this case the exec pod kube-proxy disabled the ClusterIP but the kube-proxy on the Node with the NodePort still didn't flushed the IPs ... kube-proxy's are not in sync flushing the rules


Question: var err error machine, err = as.getVirtualMachine(types.NodeName(name)) <ins> if err == cloudprovider.InstanceNotFound { return "", cloudprovider.InstanceNotFound } </ins> if err != nil { if as.CloudProviderBackoff { glog.V(2).Infof("InstanceID(%s) backing off", name)

Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug /sig azure What happened: When performing a rolling update of my nodes (i.e., cordon, drain, delete VM on Azure side) one after the other the corresponding objects of the deleted VMs are not being deleted from etcd. This affects all patch versions while it is working fine with . What you expected to happen: The objects are removed from etcd when the backing virtual machine does not exist any more. How to reproduce it (as minimally and precisely as possible): Create a Kubernetes cluster with version on Azure, cordon and drain the node and delete the backing VM afterwards. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.9.X Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): CoreOS Stable 1632.3.0 Others: machine type
It requires some time for controller manager to detect the removed VM on Azure. Could you share the logs of kube-controller-manager?
Hi thank's for taking a a look. Well, sure, it may take some time, however, it's really fast for Kubernetes clusters on Azure with versions prior than and also for clusters on other cloud providers (even with ). I have waited more than 30 minutes, but the "old" objects are still there (which leads to a weird state/view on pods, etc.). I've seen this problem only with Azure . I just reproduced the problem: At this point in time the backing VMs for and are already gone/do not exist any more. The relevant logs of the kube-controller-manager:
Thanks of providing the information. I'm checking whether there is something wrong inside cloud provider
confirmed it's a bug in azure cloud provider
Thank you for investigating. Do you already know whether the fix will be backported or only part of 1.10?
By the way, have you configured backoff with azure cloud provider? what params have you configured?
This is the relevant part of the current cloud provider configuration: Does it look good to you?
There is a problem in backoff logic which results in retrying even the VM has already removed from Azure. With the params you set, the backoff timeout will be about 40 hours. I recommend setting a smaller cloudProviderBackoffDuration (ensure 1) since this is an exponential backoff.
Okay, thank's a lot!


Question: func ValidatePolicy(policy schedulerapi.Policy) error { var validationErrors []error <ins> priorities := make(map[string]schedulerapi.PriorityPolicy, len(policy.Priorities)) </ins> for _, priority := range policy.Priorities { if priority.Weight <= 0 || priority.Weight >= framework.MaxWeight { validationErrors = append(validationErrors, fmt.Errorf("Priority %s should have a positive weight applied to it or it has overflown", priority.Name)) } <ins> validationErrors = append(validationErrors, validatePriorityRedeclared(priorities, priority)) } predicates := make(map[string]schedulerapi.PredicatePolicy, len(policy.Predicates)) for _, predicate := range policy.Predicates { validationErrors = append(validationErrors, validatePredicateRedeclared(predicates, predicate)) </ins> } binders := 0

Currently custom predicate and priorities can be created through the . The problem is that a custom predicate/priority, like ServiceAffinity, can be multiple times in the Policy with different names, possibly with different arguments. This semantic (i.e., specifying multiple instances of the same custom predicate with different arguments) is only useful if different s were specified because in all cases of custom predicates/priorities the arguments are merge-able (for example, from multiple CheckServiceAffinity custom predicates can be merged into one set). In the world of framework plugins, we don't currently have a way to maintain those semantics. to run. The question here: do we really want to maintain the old semantics of custom predicates and priorities? in other words, do we need to change the ComponentConfig API to somehow allow specifying multiple instances of the same Plugin with different arguments/weights, or should we consider that this is not supported? I always lean towards simplifying things, so my suggestion is to not support that, but I would like to know if there are actual deployments that rely on this. /sig scheduling /cc
/cc
We discussed this in the sig meeting, and put the issue for discussion on the scheduler and kubernetes-dev mailing list, I believe this behavior is not intentional, and so we should add logic to fail the scheduler configuration when a policy includes more than one custom predicate/priority of the same type. /assign
/assign
any progress on this one?
I haven't made any progress on this yet, but it's on my priority now
In migrating custom predicates to filter plugins , I am gonna ignore the custom predicate name and use the default name.


Question: limitations under the License. */ <ins> // This test uses etcd that is only fully supported for AMD64 and Linux // https://etcd.io/docs/v3.5/op-guide/supported-platform/#support-tiers </ins> package cleanup import (

periodic-kubernetes-unit-test-ppc64le TestCleanup Probably since it was merged in https://k8s- No response The error from build log trace is: Please go through the above job for complete build log. /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test has been constantly failing in our environment and not just flaking. Also, We see it still failing even after your fix for is the job that ran after merge of
^^
cc
This is a data race in the control plane setup. cc
So yes, this is different from the failure that I ran into. It's still a flake (races don''t always go wrong), but apparently the conditions in your jobs are so that it always triggers.
I'll take care of this /assign
At first glance it looks like access to in lacks protection by a mutex. But why are and racing with each other in different goroutines, and why is called first?
in some executions it seems that it just timeouts, no races I think that the problem is that this environment is under resourced and this test is not able to boot the apsierver , that may explain the race and why Stop is racing with Start
Should we perhaps make this unit test run only on amd64 architectures, with the rationale a) that it isn't architecture-dependent and thus testing it in one unit test job is sufficient and b) that it is an expensive test that stresses slower architectures too much?
the apiserver times out starting up, I think we can make it lighter with it sound fair to me, since we use etcd and it is only guaranteed to build do you want to make the honors? or should I?
Can you do it because we probably want both and you know best how to explain the patch above in the commit message?

I thought about this twice, and I don't think we should tune the tests for running on constrained environments, this is a slippery slope, this environment is clearly constrained , there are also other issues reporting timeouts in this environment ... we should not tune test just to pass The skip based on etcd not supported may be reasonable though
Met this again when running , see
this is for the ppc64le specific problem, we should open a new one for this test it seems there were 2 occurrences more
For tracking, opened a new issue here


Question: equal: false, }, { <ins> name:  "empty objects are equal", lhs:   UnstructuredToVal(map[string]interface{}{}, emptyObjectSchema), rhs:   UnstructuredToVal(map[string]interface{}{}, emptyObjectSchema), equal: true, }, { </ins> name:  "identical objects are equal", lhs:   UnstructuredToVal(map[string]interface{}{"field1": "a", "field2": "b"}, objectSchema), rhs:   UnstructuredToVal(map[string]interface{}{"field1": "a", "field2": "b"}, objectSchema),

While migrating webhooks to CEL for one of the structs in our CRD api, I noticed that the empty object in CEL is interpreted wrongly. Empty map object seems to be interpreted correctly. In the example below, both map and object are sent empty () in json output. returns true json (captured on the client side) sent to the apiserver CRD schema No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
CEL only sees the JSON sent to the server, and is correctly evaluated as present. The issue is with the client sending the request populating a value in that field. How is the request being constructed? /remove-kind bug /kind support
/cc /triage accepted
Upon investigation, the problem lies there: We requires either or be set.
Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified . It is a corner case in CRD since we normally would ask for properties defined if the field type is (I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign
GitHub didn't allow me to assign the following users: jiahui. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Thanks for raising this issue! I was debugging offline with and seems like CRD/CR validation didn't handle the situation of unsetting properties with field type specified 'Object1object`(I could not think of use case which having properties unset). But it might ok to allow it through since it won't hurt from CRD perspective. We will fix in CEL values handling which treat this as empty map passed. /assign </details
also check if the CRD validation allows the or values, and if so, if the CEL code handles those properly as well
FYI: We do not allow in crd structured schema now:


Question: <ins> // +build !race /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package master // This test file is separated from master_test.go so we would be able to disable // race check for it. TestValidOpenAPISpec will became extremely slow if -race // flag exists, and will cause the tests to timeout. import ( "net/http" "net/http/httptest" "testing" openapigen "k8s.io/kubernetes/pkg/generated/openapi" "k8s.io/kubernetes/pkg/genericapiserver" "github.com/go-openapi/loads" "github.com/go-openapi/spec" "github.com/go-openapi/strfmt" "github.com/go-openapi/validate" ) // TestValidOpenAPISpec verifies that the open api is added // at the proper endpoint and the spec is valid. func TestValidOpenAPISpec(t *testing.T) { _, etcdserver, config, assert := setUp(t) defer etcdserver.Terminate(t) config.GenericConfig.EnableIndex = true config.GenericConfig.OpenAPIConfig = genericapiserver.DefaultOpenAPIConfig(openapigen.OpenAPIDefinitions) config.GenericConfig.OpenAPIConfig.Info = &spec.Info{ InfoProps: spec.InfoProps{ Title:   "Kubernetes", Version: "unversioned", }, } config.GenericConfig.SwaggerConfig = genericapiserver.DefaultSwaggerConfig() master, err := config.Complete().New() if err != nil { t.Fatalf("Error in bringing up the master: %v", err) } // make sure swagger.json is not registered before calling PrepareRun. server := httptest.NewServer(master.GenericAPIServer.HandlerContainer.ServeMux) defer server.Close() resp, err := http.Get(server.URL + "/swagger.json") if !assert.NoError(err) { t.Errorf("unexpected error: %v", err) } assert.Equal(http.StatusNotFound, resp.StatusCode) master.GenericAPIServer.PrepareRun() resp, err = http.Get(server.URL + "/swagger.json") if !assert.NoError(err) { t.Errorf("unexpected error: %v", err) } assert.Equal(http.StatusOK, resp.StatusCode) // as json schema var sch spec.Schema if assert.NoError(decodeResponse(resp, &sch)) { validator := validate.NewSchemaValidator(spec.MustLoadSwagger20Schema(), nil, "", strfmt.Default) res := validator.Validate(&sch) assert.NoError(res.AsError()) } // Validate OpenApi spec doc, err := loads.Spec(server.URL + "/swagger.json") if assert.NoError(err) { validator := validate.NewSpecValidator(doc.Schema(), strfmt.Default) res, warns := validator.Validate(doc) assert.NoError(res.AsError()) if !warns.IsValid() { t.Logf("Open API spec on root has some warnings : %v", warns) } } } </ins>

Haven't seen this one before and I am not sure it has any relation with the that hit it. https://k8s-
From what I can gather, it is being killed due to taking over 5 minutes which is roughly how long it takes (~300s) on successful unit builds.
bumping priority. I'm seeing this pretty frequently this morning. I see OpenAPI and associate you. You know something about this one?
Yes, I wrote this test. It was failing (timing out) for quite some time on jenkins but passing quite fast locally. I've disabled it but after a while it started passing and I assumed it was a configuration change on test machines. I will spend some time on it again to see if I can find the root cause and will disable it otherwise. This test is passing locally in about 15 second. Do you have any idea why it is timing out (or passing near to 300s) on jenkins machines? where do you see the successful test stats?
I see what you do. Locally it takes 25s, with bots it is 300s. My guess is that we are constrained by CPU and all the encode/decode is causing strain either on CPU or GC. The bots fail about 75% for me so I just keep retrying until success. Just had 2 this morning pass at 296s.
Curiously, it's taking about 20 seconds with Bazel. this might be worth looking at.
I think we run tests in Jenkins with , which might explain the slowness?
Nevermind, we hit it in a bazel run: https://k8s-
-race does kill the performance, 25s -166s which is on par with bazel.
is it possible to disable -race for this test? and should we do that if it is possible?
Now blocking the submit queue.
I think I am missing something here. I've checked some of the attached PRs. Links to the pkg/master folder in master branch has the change (new file called but non of the recent PRs I checked has the file in their branch: That means they didn't sync with master? What am I missing here?
OK. I see, the tests are patching the change of the PR into the master, so everybody should get that if they retest, but it look like my test is not blocking submit queue. there should be another issue.


Question: if accountName, accountKey, err = b.util.GetAzureCredentials(b.plugin.host, b.pod.Namespace, b.secretName); err != nil { return err } <del> os.MkdirAll(dir, 0750) source := fmt.Sprintf("//%s.file.core.windows.net/%s", accountName, b.shareName) </del> <ins> os.MkdirAll(dir, 0700) source := fmt.Sprintf("//%s.file.%s/%s", accountName, getStorageEndpointSuffix(b.plugin.host.GetCloudProvider()), b.shareName) </ins> // parameters suggested by https://azure.microsoft.com/en-us/documentation/articles/storage-how-to-use-files-linux/ <del> options := []string{fmt.Sprintf("vers=3.0,username=%s,password=%s,dir_mode=0777,file_mode=0777", accountName, accountKey)} </del> <ins> options := []string{fmt.Sprintf("vers=3.0,username=%s,password=%s,dir_mode=0700,file_mode=0700", accountName, accountKey)} </ins> if b.readOnly { options = append(options, "ro") }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: }, } } <del> return f.PodClient().CreateSync(pod) </del> <ins> return pod </ins> }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: All the v1.18 / master jobs in: Which test(s) are failing: [sig-windows] Hybrid cluster network for all supported CNIs should have stable networking for Linux and Windows pods Since when has it been failing: Since this was merged: Testgrid link: Reason for failure: Previously, we've centralized several images into agnhost, including test-webserver. The Hybrid cluster network test was using the test-webserver image, and was updated to use agnhost, but without properly making it so it behaves like test-webserver, resulting in a failing test. Anything else we need to know: /sig windows /sig testing /priority important-soon


Question: FSCK=fsck.ext4 MOUNT_OPTIONS="discard,defaults" <del> MKFS="mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 -F" </del> <ins> MKFS="mkfs.ext4 -F" </ins> if [ -e /etc/redhat-release ]; then <del> if grep -q '6..' /etc/redhat-release; then # lazy_journal_init is not recognized in redhat 6 MKFS="mkfs.ext4 -E lazy_itable_init=0 -F" elif grep -q '7..' /etc/redhat-release; then </del> <ins> if grep -q '7..' /etc/redhat-release; then </ins> FSCK=fsck.xfs MKFS=mkfs.xfs fi

Here's logs from formatting a 10GB HDD disk in Azure as ext4: It took seven and a half minutes to complete . Here's me doing it manually from the node: And even from a container, somewhat emulating kubelet: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu (Xenial) 16.10 Kernel (e.g. ): Linux 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubernetes-anywhere / static pods with kubelet / kubelet in container What happened: It took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred What you expected to happen: It to take a matter of seconds, similar to how it does if I execute the command manully. How to reproduce it (as minimally and precisely as possible): Build with on top. Deploy a cluster. Try to attach an unformatted VHD. Observe along with watching logs and confirm it really does take that long. Anything else do we need to know: I'm running containerized. Like this:
Can you confirm you were not seeing this behavior during your testing? I assume you were not running containerized?
Even if I and execute the same command from there it executes in a matter of a few seconds.
I'm seeing this behavior even when running as a normal systemd unit, and not as a docker container under a unit. Here's : And I'm still seeing the very slow , so it seems not related to running kubelet in container. I have also tested to ensure that this isn't something some sort of host caching. Even on another blank disk being attached for the the first time, executes quickly except when run by .
I confirm this is this case on my setup running local-up- I have a 100GB HDD disk. It took 30 seconds to format on the console but 2 minutes by kubelet. Is there any I/O constraint imposed on kubelet?
to see if she knows why would be slow when called from the kubelet.
cc/ Who is working on node performance benchmark.

Can we quickly run an experiment on GCE to see if we can reproduce the issue. Let's first rule the impact of containerized Kubelet. If the issue is still exist, I suspect the issue in Kubelet mounting logic. We can enable some tracing, and find the bottleneck.
I tried formatting a 10GB HDD in GCE Kubernetes (not containerized). The time of formatting in kubelet log is ~1s. The log is:
On Azure, a 10GB disk took 18s to mkfs by kublet but 4s to mkfs on the console. Do you have a chance to test these two scenarios? Size probably is not a critical factor for gce since gce pd is provisioned at fixed IOPS per GB, while Azure (also AWS) disks are provisioned at fixed IOPS per volume.
There's more variables in play or else I'm seeing a different edge case than you are, On Azure with a cluster, a 10GB disk for me consistently takes 7+ minutes. (, standard storage, HDD type). It was happening with a and with premium storage and SSD type data disk too... I can make a cluster available to anyone who would like to investigate the exact behavior I've been observing. I can provide kubeconfig+ssh creds. Contact me on Slack (colemickens) if interested.
mkfs's kernel stack after started by kubelet is always as the following. Not sure if it related to blkio cgroup
I have actually created a branch of to give me a cluster where is never running in a container and I'm still seeing the exact same issue. Took 7.5 minutes again, based on kubelet logs and watching . I think we can safely eliminate "containerized kubelet" as being the source of the problem.
Though if I remember correctly, kubelet places itself in a cgroup when it starts up. ( can confirm) so it's possible that it is still due to cgroup I/O throttling.
Above test result proves my initial suspect that the root cause in 7-min use case is containerized kubelet. But recent experiment is not running in a container.

could you please provide us more information: 1) cat /proc/<kubelet-pid/cgroups to find kubelet's cgroup 2) check & blkio.weightdevice setting 3) ps ax | grep makfs to find process id, the find which cgroup the process belongs to 4) check 3) cgroup's and blkio.weight_device If both 2) and 4) don't have issues with cgroup setting, please check if there is parent cgroup for them, and their setting since both CFQ and throttling supports hierarchy.
FYI: Kubelet does not perform any block IO throttling as of now for itself or any other entity on the node. , Dawn Chen wrote:
Yes, we know Kubelet doesn't. But the issue here indicates containerized Kubelet's IOs is throttled, and we are trying to figure out why, especially the user is running kubernetes-anywhere on Azure, which we, node-team don't normally test.
Here's how I was running originally: Here's how I was running when testing it not in a container: I will try to collect the data you mention but I'm going to have to learn how to do some of it along the way, I don't know much about cgroups.
this is not limited to azure, saw it on aws as well.
it seems only the hit only happened to ext4, xfs didn't have this problem.
I ran across the same performance issues and tried to investigate. I use Azure Dynamic Disk provisioning to provision a 1TB volume and it takes very long time for mkfs to finish. Doing the "mkfs.ext4 -F -E lazyitableinit=0,lazyjournalinit=0 /dev/sdd" manually is also very slow, but I did not compare the performance to the call done from inside kubelet. While running the mkfs manually, I observed that iotop shows around 62mb/s and that the free memory immediately shrinks from 2000mb to 100mb. It looks like mkfs is using up all free memory for the buffer cache. I even observed oom situations in the kernel log and processes being killed because of this. I've seen this behavior while running mkfs manually and while it was run by kubelet.

Removing the extended flag "lazyitableinit=0" from the manual mkfs call fixes both problems, performance and memory/buffer cache usage. My solution will now be to use xfs (as rootfs reported it to work better). But IMHO the lazyitableinit should be completely removed from the mkfs call.
Can we try to make sure this gets addressed in 1.6?
I did some further investigation here and it turns out there are other factors involved as well, especially on Azure. My initial statement that I got OOM due to mkfs.ext4 seems to be not true, as the OOMs were caused by fluentd-elasticsearch. I tried to use mkfs.ext4 with "-E discard=1" as I assumed that it should speed up things dramatically. This did not make a difference however. While trying to understand the mke2fs code from e2fsprogs, I realized that discard is enabled by default and always performed as first step when the underlying block device supports this. From the mkfs.ext4 logs you can actually see that it is executed at the beginning. It turns out that mke2fs does only skip inode table initialization when the kernel reports that a discard also ensures zeroing of the data. mke2fs checks this by doing an ioctl to BLKDISCARDZEROES. Support can also be checked with On Azure however, the kernel reports that a discard does NOT zero out the discarded blocks. This results in a loop, using normal write calls to write a lot of zeros to the block dev. This completely fills up the page cache with dirty pages, slowing down the whole system extremely. If there is not much free memory available that can be used for write buffering, the write rate goes down to very low values as well. I did not check the kernel code for lazyitableinit, but I would assume that it uses much more efficient code to write the zeros to the block device (e.g. by not relying on dirty pages). I'd also assume that it does these checks for the discard+zeroout behavior/support and then use discard if possible. If my assumption is correct, then it should not affect performance in a bad way when lazy init would be enabled again. The kernel would then do the discards in a lazy fashion, but finish pretty fast.

On Azure, the lazy init would still take some time, but use much less system resources (page cache) as the non-lazy solution. And most importantly, the device can already be mounted after a few seconds. (UPDATE: please check next comment) I tried to figure out if a discard (using the blkdiscard cmd) does really leave you with non-zeroed data and it looks like it actually becomes zero. I can however not see if this happens immediately or with a delay. Also, in the Azure Portal I can see that the "Billable Size" goes from several GB back to 0 after a short time. My assumption is that on other cloud providers, e.g. GCE and AWS, the block devices report that discard zeroes out the data. Can anyone with a GCE/AWS machine check this with the cat call from above? This would explain why this slowness does not occur on non-Azure clouds. Can you check why the kernel reports that discard does not zero-out? Is this a bug in Azure maybe?
~Ok, my assumption about the kernel using discard for lazy init was wrong. Just checked the kernel source and it simply issues multiple write bios which write zero-pages. So it would definitely have a negative impact on other cloud providers when lazyitableinit would be enabled.~ Would it be a viable solution to check support for BLKDISCARDZEROES in kubernetes before doing the mkfs and then decide based on that if lazyitableinit should be enabled or not?
Argh, digging deeper into the kernel turned out that it actually uses discard for zeroing out (check and ). So my initial assumption was correct. Sorry for all the go home now :)
Just checked the discard support on AWS and it is not supported at all, so it always does a normal write to clear/init the inode tables.
here's the output from a GCP instance: We haven't seen any slowness on GCP, even though the above settings are identical to our Azure instance's. We're trying to solve the same slowness issue on Azure, just not with kubernetes :) (we're Cloud Foundry). We're not using containers on this particular VM. Also, the problem manifests with but not .

Related issue here:
We discovered that by using instead of the problem went away — disks which previously took 8+ minutes to mkfs.ext4 took seconds. Alternatively, will also work as a partitioner (i.e. the subsequent will finish within seconds). Really. We're not sure why the tool we use to write one sector (i.e. the MBR) on a disk would make such a difference. HTH,
Did you discover this by manually running / on the nodes, or did you actually modify to use ?
This sounds like alignment issues. As far as I remember, sfdisk does not do proper alignment and thus may result on very slow performance when you are not on classical spinning disks. Azure disk provisioner does not even create partitions on the disks and thus alignment should not be the problem here. Just tested creating a partition with sfdisk and iotop showed only 7-10mb/s throughput when doing mkfs. Not using a partition at all or using fdisk gives around 60mb/s.
: We discovered this by manually running gparted / sfdisk on the instances we deployed on Azure (i.e. we're not using containers). We don't use Kubernetes — we are developers on Cloud Foundry, which is an alternative to Kubernetes. When we ran into this problem, we were pleased to discover we were not the only ones who have been bitten by this bug. And we wanted to make sure our Kubernetes kin were aware of the fix that we found to be effective. some notes on our research (and I cannot thank you enough for your detailed descriptions): we also found throughput of 7-10MB/s when was healthy when things were unhealthy, showed throughput of ~300kB/s during . we found that by passing (use direct IO and bypass buffer cache), we doubled our performance to ~700kB/s, but it was still ridiculously long (3 minutes).
I am going to try to do some additional investigation tonight, but I'm unsure of how to translate your observations to Kubernetes here. As implied, we don't use anywhere here (that I'm aware of). There's three scenarios: -dynamically provisioned. User brings a pre-formatted VHD. Everything goes smoothly as the merely needs to mount the block device and tell to mount it into the container. -dynamically provisioned.

User attaches an empty disk to an Azure VM, detaches it, uses it in a normal AzureDisk volume on a Pod. This is the scenario in the original post, and you can see that it is being mkfs'd without any partitions. disk provisioning. The VHD footer is written into the page blob, and then formats the whole disk, unpartitioned (trusting description of the provisioner here). I'm hesitant to rule out an alignment issue since you're seeing similar symptoms as we are here, but I'm not sure how to reconcile that with my other points in this comment...
What do you think about It would solve this issue. It would not tell us why it is slower in kubelet, but the problem itself would be solved. Should I CC more people to get this discussed faster? I would even like to see this in an upcoming 1.5.x release.
a lead Azure Engineer ( ) found the problem — it is an alignment problem gone horribly wrong: In situation like unaligned partition boundary, a big I/O request (e.g. a 32KB write) can be split into 64 512-byte writes by the kernel block I/O subsystem in the worst case, easily hitting the 500-IOPS throttling of Azure. It was made to simplify Hyper-V storage driver’s code, so Hyper-V’s storage driver picked up this patch in kernel versions: 4.4+, that is why only Azure is impacted. The community member who submitted the original patch already for this issue, waiting for Linux community to merge.
Does that also affect cases where the disk is unpartitioned, ?
If I understand the description correctly, this only applies to cases where unaligned partitions are in use. This is not the case for Azure Dynamic Disk provisioning and also for all the tests performed so far by all the participants here. As noted in , I did some manual tests with partitioning. And having unaligned partitions really degrades performance a lot (reducing mkfs throughput from 60mb/s to 10mb/s). But as said, we don't have unaligned partitions when using Azure Dynamic Disk actually don't have any partitions at all as we create the FS directly on the block device (e.g. /dev/sdc instead of /dev/sdc1). The real issue here is that lazy inode table initialization is disabled for mkfs.ext4.

The reason for disabling it can be found here: The assumption (BLKDISCARD+BLKDISCARDZEROES support) made in this commit is not true for all cloud providers, e.g. Azure and at least for AWS magnetic disks. I did some calculations to figure out how much zeroes must be written by mkfs.ext4 to initialize the inode tables for a 1TB disk. If you check "tune2fs -l /dev/sdX" you'll see that there are inodes with a size of 256 bytes allocated for a 1TB file system. This gives a total of 16G! These 16G of zeroes are written by mkfs.ext4 when lazy init is disabled, in a very very naive way. Writing 16G of zeroes already takes 4:30 minutes when you have a throughput of 60mb/s. Now if there is something that throttles the troughput (and it looks like kubelet does this somehow), it gets even worse. And that's not all, while this is written do disk, the complete system suffers on low memory due to all of it being used for write caching/buffering. If other processes want to allocate memory in this situation, they're blocked until enough from the buffers is flushed to disk. Please check It fixes all this by simply re-enabling lazy inode table initialization. I also tried to describe why this change should not affect cloud providers that previously thought that disabling lazy init would be an improvement.
I don't believe the partition alignment issue would affect cases where the disk is unpartitioned (i.e. using the entire disk, e.g. ). I suspect that any use of an entire raw device would be properly aligned (any filesystem that starts on sector 0 of the hard disk would be properly aligned), but that's a hunch and nothing more. : I think your assessment is on target — the problem we (Cloud Foundry) are seeing is caused by a partition misalignment, but it seems that the problem described in this thread has a different cause. We currently use a lazy inode table initialization, so we're covered on that front:

According to : While the is exactly the opposite. This significantly slows down large ext4 volume creation.
This might be worth bringing up at the storage f2f coming up.I am going to add to the agenda. This is an interesting approach to what I think may be a few larger issues. Questions and thoughts: What is a good SLO for mkfs? If the SLO is going to be violated due to size should k8s do something or should the user pre-format? There are tradeoffs for initting the filesystem lazy or not. Supporting all the options for a user may require an API change. Is this a persistent volume only thing?
I may miss something, but on a random GCE node with 500GiB magnetic(!) disk and Fedora 24 takes around 2 seconds. ext3 is a different story, it takes 74 seconds.
I cannot imagine any reason why anyone would be installing ext2/ext3 on a 500GB drive. If they do they should be expecting things to be slow. In any case knowing what the SLO is is still not a crazy idea.
This is probably not a big problem with GCE, since GCE PD's IOPS is per GB. This is mostly a problem for EBS/Cinder/Azure, since the IOPS per volume is fixed.
If we're going to do this at all, we should really think hard about this problem top-to-bottom. Ext3 vs ext4 vs other FS? Myriad filesystem options, etc.
I got bitten by this when implementing e2e tests for all supported storage class options on AWS. "Cold HDD" volume type has max. 250 IOPS and the smallest one is 500 GB. It takes 3-4 minutes just to format the volume, which slows down the test quite considerably. Was there any conclusion on f2f meeting? was in commit , which claims to be rewritten to Go. Original script not exist any more on the link above. Was there some smart idea in original ?
see some inconclusive investigation at
Please take a look at I encountered not only performance issue due to lazyitableinit=0, but also oom situations and thus killed processes. Due to the amount of data written to the disk, all free memory gets filled with dirty buffer cache.


Question: } ` <ins> var constructorWithTypeMeta = ` // $.ApplyConfig.ApplyConfiguration|public$ constructs an declarative configuration of the $.ApplyConfig.Type|public$ type for use with // apply. func $.ApplyConfig.Type|public$() *$.ApplyConfig.ApplyConfiguration|public$ { b := &$.ApplyConfig.ApplyConfiguration|public${} b.WithKind("$.ApplyConfig.Type|singularKind$") b.WithAPIVersion("$.APIVersion$") return b } ` </ins> var constructor = ` // $.ApplyConfig.ApplyConfiguration|public$ constructs an declarative configuration of the $.ApplyConfig.Type|public$ type for use with // apply.

Most of the FooApplyConfiguration constructors pre-fill the Kind and APIVersion fields to make using them smooth. But () is missing this, making patterns like this fail: Currently this code fails with (server-side) error . The constructor should set the minimum required fields for the resulting object to be usable with calls, similar to how e.g. the `D Create a deployment (however) Create a clientset, and get the Deployments client from it Try to call deployments.ApplyScale using a ScaleApplyConfiguration created by the Scale() constructor I can see that the presence or absence of these metadata fields is driven by some code generation, but I wasn't able to follow through to what configures that code generation. I assume it's something in the schemas somewhere, but that wasn't clear to me. <details</details<detailsminikube v1.25.1 </detailsN/A N/A N/A N/A
/sig api-machinery
/assign /triage accepted
/assign


Question: Description: `By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs.` + "nn"}}, }, <ins> {"e2e/foo.go", ` var _ = framework.KubeDescribe("Feature", func() { Context("with context and extra spaces before It block should still pick up Testname", func() { //					Testname: Test with spaces //Description: Should pick up testname even if it is not within 3 spaces //even when executed from memory. framework.ConformanceIt("should work", func() {}) }) })`, []conformanceData{{URL: "https://github.com/kubernetes/kubernetes/tree/master/e2e/foo.go#L8", TestName: "Test with spaces", Description: `Should pick up testname even if it is not within 3 spaces even when executed from memory.` + "nn"}}, }, </ins> } func TestConformance(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When the comment section associated with the conformance test is separated with more that four spaces/tabs etc then the comment is ignored. What you expected to happen: Comment associated with the conformance test should be used instead of ginkgo comments. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Received output (due to extra tabs) Expected output: Release : v1.13 If the restart policy is set to ‘Always’, Pod MUST be restarted when terminated, If restart policy is ‘OnFailure’, Pod MUST be started only if it is terminated with non-zero exit code. If the restart policy is ‘Never’, Pod MUST never be restarted. All these three test cases MUST verify the restart counts accordingly. Environment: Kubernetes version (use ): 1.13 /kind bug
/area conformance /sig architecture thanks for the report and the fix!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten