Question: if ipFamily, err := getIPFamilyFromIP(ip); err == nil { ipFamilyMap[ipFamily] = append(ipFamilyMap[ipFamily], ip) } else { <del> klog.ErrorS(nil, "Skipping invalid IP", "ip", ip) </del> <ins> // this function is called in multiple places. All of which // have sanitized data. Except the case of ExternalIPs which is // not validated by api-server. Specifically empty strings // validation. Which yields into a lot of bad error logs. // check for empty string if len(strings.TrimSpace(ip)) != 0 { klog.ErrorS(nil, "Skipping invalid IP", "ip", ip) } </ins> } } return ipFamilyMap

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As we update Kube-proxy to version 1.21, we observe a stream of following error entries within the Kube-proxy logs: Skipping invalid IP: ". As we do not see anything wrong with the EKS cluster itself, this message should not be reported as an error. an EKS cluster v1.21 and launched a managed nodegroup within this cluster. use the managed add-on or manage Kube-proxy yourself. Either way, the EKS provided kube-proxy image (kube-proxy:v1.21.2-eksbuild.2) is based on the upstream image and this error can be traced to the upstream code as I will be outlining in the next section. logs for the kube-proxy pod: kubectl logs -n kube-system <kube-proxy-pod-nameIn my replication, I could analyze this: These errors were recurring in the kube-proxy logs every 15 minutes which I concluded after watching the log stream for a 1+ hour window. To be able to make sense of the logs better, I increased verbosity of kubeproxy to 9: kubectl patch ds kube-proxy -n kube-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command/1", "value":"--v=9"}]' In the logs, I could notice the following snippet recurring periodically (every 15 minutes): This hinted that a 'resync' was behind causing this errors being pushed every 15 minutes, however, before diving deeper into the root cause there, I searched for the origin of 'Skipping invalid IP: ' line and tried to reverse engineer back to the root cause for these errors. For this, I went through the '' in kube-proxy upstream github (release-1.21 branch). I started with the 'MapIPsByIPFamily' function which is where this error is handled and thrown [1]. This error is returned to it by a child function getIPFamilyFromIP [2]. The purpose of this function is to identify if the IP string passed is IPv6 or IPv4.

So, I backtracked a bit from the 'MapIPsByIPFamily' function and observed that it was referenced on line 186 by 'newBaseServiceInfo' in the file of kube-proxy [3]. Here I noticed that 'service.Spec.ExternalIPs' is passed as string to the 'MapIPsByIPFamily' function. Now, not all services have an ExternalIP defined in their spec. This is when an empty string '' is passed to the latter and hence the snippet : [klog.Errorf("Skipping invalid IP: %s", ip)] translates to 'Skipping invalid IP: ' in the actual error message. However, in the following error handling block: Rather than simply logging an error, there could be a check for this case of an empty string (as this is pretty common for clusters to have services without ExternalIP defined). If it is not possible to handle this error in this snippet for some reason, the verbosity level could be highered to atleast 4 so that the default kube-proxy image that runs at level 2 generally is not bombarded with a stream of recurring errors that are not actually errors. Can you please help fix this issue? [1] [2] [3] Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: EKS with EC2 worker nodes OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
/triage accepted nice report thanks /cc
Thanks. is your service defined with ?
I performed a 'kubectl get svc <svc_name-o yaml' and I could confirm that my services do not have an explicit .Spec.ExternalIPs = {""}. The output for one of them was as follows:
/assign
There was a similar problem with blank spaces too on the field , in this case the IPs were invalid because a blank space was appended to the IP If is not an ExternalIPs, then it has to be in the Status.LoadBalancer.Ingress.IP I think that if the function errors, we should return the error instead of logging inside, so the caller can handle it and raise a meaningful error, pointing the user to the specific service and field , what do you think?
iirc we have to ignore and log the error. Because there some cloud providers that sets this as " ".

And since there is no validation in place for that, we can not afford to add validation now (break the api). What i am looking into is limit the amount of times we log this error on proxy or remove it at all.
yeah, validation train has left the station some time ago :smile: I was trying to mean that the error should point to the service causing the issue, so users can act on it ...
Thanks for taking a look guys, really appreciate your help with this I could see that the fix has been planned as part of v1.23 milestone. Can you please clarify if this fix can/would be backported to previous Kubernetes versions and if there is a timeline for the same. This will help plan if we will continue to see such errors for some time now. Thanks again, good day !


Question: dc.podLister = podInformer.Lister() dc.podListerSynced = podInformer.Informer().HasSynced <del> pdbInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> pdbInformer.Informer().AddEventHandler( </ins> cache.ResourceEventHandlerFuncs{ AddFunc:    dc.addDb, UpdateFunc: dc.updateDb, DeleteFunc: dc.removeDb, }, <del> 30*time.Second, </del> ) dc.pdbLister = pdbInformer.Lister() dc.pdbListerSynced = pdbInformer.Informer().HasSynced

/area controller-manager What would you like to be : Don't resync disruption in disruption controller. Why is this needed: There are 3000+ deployments in my k8s cluster, thus when i create all deployments,the depth of disruption workquque is too long in every period, the period of collecting data is 10s, i found that the depth of disruption workquque less than or equal the number of disruption , and there is also a AddEventHandlerWithResyncPeriod(30s) to resync all disruption, I don't know why there need to have to AddEventHandlerWithResyncPeriod all disruption? Can we not resync disruption and just watch it's changes?
/sig apps
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: for _, disk := range disks { for _, diskName := range diskNames { <del> if disk.Name != nil && diskName != "" && *disk.Name == diskName { </del> <ins> if disk.Name != nil && diskName != "" && strings.EqualFold(*disk.Name, diskName) { </ins> attached[diskName] = true } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: // Note: `fsizes` values are enforced to each be at least `MinFileSize` and a multiple of `MinFileSize` //   bytes. func testVolumeIO(f *framework.Framework, cs clientset.Interface, config framework.VolumeTestConfig, volsrc v1.VolumeSource, podSecContext *v1.PodSecurityContext, file string, fsizes []int64) (err error) { <del> dir := path.Join("/opt", config.Prefix, config.Namespace) ddInput := path.Join(dir, "dd_if") </del> <ins> ddInput := filepath.Join(mountPath, fmt.Sprintf("%s-%s-dd_if", config.Prefix, config.Namespace)) </ins> writeBlk := strings.Repeat("abcdefghijklmnopqrstuvwxyz123456", 32) // 1KiB value loopCnt := testpatterns.MinFileSize / int64(len(writeBlk)) // initContainer cmd to create and fill dd's input file. The initContainer is used to create

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] volumeIO should write files of various sizes, verify size, validate content [Slow] Since when has it been failing: 11/13 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: This is consistently failing in 1.13 branch and is release blocking. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test /sig storage /priority critical-urgent /milestone v1.13 /assign /cc
The test case is failing because it wrote 2MB instead of the expected 1MB. It's also very odd that the test is failing in soak, but passing in regular CI suites.
There's a few problems: .GetVolumeSource returns "/tmp" for everything. This means all tests will share the same directory. This should be probably use some unique directory instead volume IO test generates a unique mountPath directory, however that doesn't mean a unique directory is actually created on the host. Ie, a mountPath of "/a/b/c" on a hostpath volume at "/tmp" doesn't mean directory "/tmp/a/b/c" is created. io test file name is not unique seems like the IO test is not cleaning up the files it creates, so the files are getting reused on subsequent tests due to the non-unique naming issues.
The other thing I can't explain is why it's failing only now
Fix is here:
As per discussion in 1.13 burndown this morning, this is a test job issue and its not release blocking at this point. So we can take this fix in 1.14 as opposed now during code freeze let us know if you feel otherwise
/milestone v1.14 /priority important-soon
/remove-priority critical-urgent


Question: // namespace used for all tests, do not change this const testNamespace = "etcdstoragepathtestnamespace" <ins> // allowMissingTestdataFixtures contains the kinds expected to be missing serialization fixtures API testdata directory. // this should only contain custom resources and built-in types with open issues tracking adding serialization fixtures. // Do not add new built-in types to this list, add them to k8s.io/api/roundtrip_test.go instead. var allowMissingTestdataFixtures = map[schema.GroupVersionKind]bool{ // TODO(https://github.com/kubernetes/kubernetes/issues/79027) gvk("apiregistration.k8s.io", "v1", "APIService"):     true, gvk("apiregistration.k8s.io", "v1beta", "APIService"): true, // TODO(https://github.com/kubernetes/kubernetes/issues/79026) gvk("apiextensions.k8s.io", "v1beta1", "CustomResourceDefinition"): true, gvk("apiextensions.k8s.io", "v1", "CustomResourceDefinition"):      true, // Custom resources are not expected to have serialization fixtures in k8s.io/api gvk("awesome.bears.com", "v1", "Panda"):    true, gvk("cr.bar.com", "v1", "Foo"):             true, gvk("random.numbers.com", "v1", "Integer"): true, gvk("custom.fancy.com", "v2", "Pant"):      true, } </ins> // TestEtcdStoragePath tests to make sure that all objects are stored in an expected location in etcd. // It will start failing when a new type is added to ensure that all future types are added to this test. // It will also fail when a type gets moved to a different location. Be very careful in this situation because

Improve the to ensure that the expected storage version for a resource is one understood by previous releases. We have API testdata from prior versions available in When checking each group/version/kind in the test: accumulate all the versions of the group/kind available in previous (non-HEAD) releases if there is a beta or stable version of the group/kind available in previous releases, then the storage version must be in the set of previously available versions It is too easy to accidentally change the storage version prematurely and edit the test/integration/etcd test to match the bug cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: !IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, err) } <del> var hcNames []string </del> <ins> clusterID, err := GetClusterID(c) if err != nil { retErr = fmt.Errorf("%vn%v", retErr, err) return } hcNames := []string{gcecloud.MakeNodesHealthCheckName(clusterID)} </ins> hc, getErr := gceCloud.GetHttpHealthCheck(loadBalancerName) if getErr != nil && !IsGoogleAPIHTTPErrorCode(getErr, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, getErr)

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package autoregister import ( "fmt" "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clienttesting "k8s.io/client-go/testing" "k8s.io/client-go/tools/cache" "k8s.io/client-go/util/workqueue" "k8s.io/kube-aggregator/pkg/apis/apiregistration" "k8s.io/kube-aggregator/pkg/client/clientset_generated/internalclientset/fake" listers "k8s.io/kube-aggregator/pkg/client/listers/apiregistration/internalversion" ) func newAutoRegisterManagedAPIService(name string) *apiregistration.APIService { return &apiregistration.APIService{ ObjectMeta: metav1.ObjectMeta{Name: name, Labels: map[string]string{AutoRegisterManagedLabel: string("true")}}, } } func newAutoRegisterManagedModifiedAPIService(name string) *apiregistration.APIService { return &apiregistration.APIService{ ObjectMeta: metav1.ObjectMeta{Name: name, Labels: map[string]string{AutoRegisterManagedLabel: string("true")}}, Spec: apiregistration.APIServiceSpec{ Group: "something", }, } } func newAPIService(name string) *apiregistration.APIService { return &apiregistration.APIService{ ObjectMeta: metav1.ObjectMeta{Name: name}, } } func checkForNothing(name string, client *fake.Clientset) error { if len(client.Actions()) > 0 { return fmt.Errorf("unexpected action: %v", client.Actions()) } return nil } func checkForCreate(name string, client *fake.Clientset) error { if len(client.Actions()) == 0 { return nil } if len(client.Actions()) > 1 { return fmt.Errorf("unexpected action: %v", client.Actions()) } action := client.Actions()[0] createAction, ok := action.(clienttesting.CreateAction) if !ok { return fmt.Errorf("unexpected action: %v", client.Actions()) } apiService := createAction.GetObject().(*apiregistration.APIService) if apiService.Name != name || apiService.Labels[AutoRegisterManagedLabel] != "true" { return fmt.Errorf("bad name or label %v", createAction) } return nil } func checkForUpdate(name string, client *fake.Clientset) error { if len(client.Actions()) == 0 { return nil } if len(client.Actions()) > 1 { return fmt.Errorf("unexpected action: %v", client.Actions()) } action := client.Actions()[0] updateAction, ok := action.(clienttesting.UpdateAction) if !ok { return fmt.Errorf("unexpected action: %v", client.Actions()) } apiService := updateAction.GetObject().(*apiregistration.APIService) if apiService.Name != name || apiService.Labels[AutoRegisterManagedLabel] != "true" || apiService.Spec.Group != "" { return fmt.Errorf("bad name, label, or group %v", updateAction) } return nil } func checkForDelete(name string, client *fake.Clientset) error { if len(client.Actions()) == 0 { return nil } for _, action := range client.Actions() { deleteAction, ok := action.(clienttesting.DeleteAction) if !ok { return fmt.Errorf("unexpected action: %v", client.Actions()) } if deleteAction.GetName() != name { return fmt.Errorf("bad name %v", deleteAction) } } return nil } func TestSync(t *testing.T) { tests := []struct { name               string apiServiceName     string addAPIServices     []*apiregistration.APIService updateAPIServices  []*apiregistration.APIService addSyncAPIServices []*apiregistration.APIService delSyncAPIServices []string expectedResults    func(name string, client *fake.Clientset) error }{ { name:               "adding an API service which isn't auto-managed does nothing", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{newAPIService("foo")}, updateAPIServices:  []*apiregistration.APIService{}, addSyncAPIServices: []*apiregistration.APIService{}, delSyncAPIServices: []string{}, expectedResults:    checkForNothing, }, { name:               "adding one to auto-register should create", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{}, updateAPIServices:  []*apiregistration.APIService{}, addSyncAPIServices: []*apiregistration.APIService{newAPIService("foo")}, delSyncAPIServices: []string{}, expectedResults:    checkForCreate, }, { name:               "duplicate AddAPIServiceToSync don't panic", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{newAutoRegisterManagedAPIService("foo")}, updateAPIServices:  []*apiregistration.APIService{}, addSyncAPIServices: []*apiregistration.APIService{newAutoRegisterManagedAPIService("foo"), newAutoRegisterManagedAPIService("foo")}, delSyncAPIServices: []string{}, expectedResults:    checkForNothing, }, { name:               "duplicate RemoveAPIServiceToSync don't panic", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{newAutoRegisterManagedAPIService("foo")}, updateAPIServices:  []*apiregistration.APIService{}, addSyncAPIServices: []*apiregistration.APIService{}, delSyncAPIServices: []string{"foo", "foo"}, expectedResults:    checkForDelete, }, { name:               "removing auto-manged then RemoveAPIService should not touch APIService", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{}, updateAPIServices:  []*apiregistration.APIService{newAPIService("foo")}, addSyncAPIServices: []*apiregistration.APIService{}, delSyncAPIServices: []string{"foo"}, expectedResults:    checkForNothing, }, { name:               "create managed apiservice without a matching request", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{newAPIService("foo")}, updateAPIServices:  []*apiregistration.APIService{newAutoRegisterManagedAPIService("foo")}, addSyncAPIServices: []*apiregistration.APIService{}, delSyncAPIServices: []string{}, expectedResults:    checkForDelete, }, { name:               "modifying it should result in stomping", apiServiceName:     "foo", addAPIServices:     []*apiregistration.APIService{}, updateAPIServices:  []*apiregistration.APIService{newAutoRegisterManagedModifiedAPIService("foo")}, addSyncAPIServices: []*apiregistration.APIService{newAutoRegisterManagedAPIService("foo")}, delSyncAPIServices: []string{}, expectedResults:    checkForUpdate, }, } for _, test := range tests { fakeClient := fake.NewSimpleClientset() apiServiceIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}) c := autoRegisterController{ apiServiceClient:  fakeClient.Apiregistration(), apiServiceLister:  listers.NewAPIServiceLister(apiServiceIndexer), apiServicesToSync: map[string]*apiregistration.APIService{}, queue:             workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "autoregister"), } for _, obj := range test.addAPIServices { apiServiceIndexer.Add(obj) } for _, obj := range test.updateAPIServices { apiServiceIndexer.Update(obj) } for _, obj := range test.addSyncAPIServices { c.AddAPIServiceToSync(obj) } for _, objName := range test.delSyncAPIServices { c.RemoveAPIServiceToSync(objName) } c.checkAPIService(test.apiServiceName) //compare the expected results err := test.expectedResults(test.apiServiceName, fakeClient) if err != nil { t.Errorf("%s %v", test.name, err) } } } </ins>

The unit tests described here are really hard to read and they've been flaking. A pattern based on indexers like this is easier to understand. I would suggest writing a new unit test file and once the coverage is consistent, just deleting the old one. We need coverage for: ignoring APIServices that aren't auto managed when they are mutated create an APIService in response to AddAPIServiceToSync delete an APIService in response to RemoveAPIServiceToSync duplicate RemoveAPIServiceToSync don't panic duplicate AddAPIServiceToSync don't panic AddAPIServiceToSync an APIService, then having the user take ownership (remove annotation), then RemoveAPIServiceToSync should not touch the APIService creating an APIService with the when no such service has been AddAPIServiceToSync, should result in APIService deletion AddAPIServiceToSync an APIService, then modify the APIService. The APIService should be stomped back to the AddAPIServiceToSync value.
hi, , about follow the indexers pattern to refactor , so I using a to replace previous and , and using to manipulate ? I wrote a basic framwork of new unit test file, is this right ? here is my code(not finished yet) :


Question: "net/url" "strings" "testing" <del> "golang.org/x/net/html" </del> ) func parseURLOrDie(inURL string) *url.URL {

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: serviceAddress := net.JoinHostPort(serviceIP, strconv.Itoa(servicePort)) for _, pausePod := range pausePods.Items { <del> sourceIP, execPodIP := execSourceipTest(pausePod, serviceAddress) </del> <ins> sourceIP, execPodIP := execSourceIPTest(pausePod, serviceAddress) </ins> ginkgo.By("Verifying the preserved source ip") framework.ExpectEqual(sourceIP, execPodIP) }

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: func (f *flexVolumeUnmounter) TearDownAt(dir string) error { <del> if pathExists, pathErr := util.PathExists(dir); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := util.PathExists(dir) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", dir) return nil } <ins> if pathErr != nil && !util.IsCorruptedMnt(pathErr) { return fmt.Errorf("Error checking path: %v", pathErr) } </ins> call := f.plugin.NewDriverCall(unmountCmd) call.Append(dir) _, err := call.Run()

/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!


Question: v1.ResourceCPU:    resource.MustParse("100m"), v1.ResourceMemory: resource.MustParse("100Mi"), }, <del> }) </del> <ins> }, node) </ins> nonCriticalBurstable := getTestPod(false, burstablePodName, v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("100m"), v1.ResourceMemory: resource.MustParse("100Mi"), }, <del> }) nonCriticalBestEffort := getTestPod(false, bestEffortPodName, v1.ResourceRequirements{}) </del> <ins> }, node) nonCriticalBestEffort := getTestPod(false, bestEffortPodName, v1.ResourceRequirements{}, node) </ins> criticalPod := getTestPod(true, criticalPodName, v1.ResourceRequirements{ // request the entire resource capacity of the node, so that // admitting this pod requires the other pod to be preempted Requests: getNodeCPUAndMemoryCapacity(f), <del> }) _, err = f.ClientSet.SchedulingV1().PriorityClasses().Create(systemCriticalPriority) framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true, "failed to create PriorityClasses with an error: %v", err) </del> <ins> }, node) </ins> // Create pods, starting with non-critical so that the critical preempts the other pods. f.PodClient().CreateBatch([]*v1.Pod{nonCriticalBestEffort, nonCriticalBurstable, nonCriticalGuaranteed})

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: } func TestServiceRegistryMakeLinkVariables(t *testing.T) { <del> service := api.Service{ JSONBase:      api.JSONBase{ID: "foo-bar"}, Selector:      map[string]string{"bar": "baz"}, ContainerPort: util.IntOrString{Kind: util.IntstrString, StrVal: "a-b-c"}, } </del> registry := registrytest.NewServiceRegistry() registry.List = api.ServiceList{ <del> Items: []api.Service{service}, </del> <ins> Items: []api.Service{ { JSONBase: api.JSONBase{ID: "foo-bar"}, Selector: map[string]string{"bar": "baz"}, Port:     8080, Protocol: "TCP", }, { JSONBase: api.JSONBase{ID: "abc-123"}, Selector: map[string]string{"bar": "baz"}, Port:     8081, Protocol: "UDP", }, { JSONBase: api.JSONBase{ID: "q-u-u-x"}, Selector: map[string]string{"bar": "baz"}, Port:     8082, Protocol: "", }, }, </ins> } machine := "machine" vars, err := GetServiceEnvironmentVariables(registry, machine) if err != nil { t.Errorf("Unexpected err: %v", err) } <del> for _, v := range vars { if !util.IsCIdentifier(v.Name) { t.Errorf("Environment variable name is not valid: %v", v.Name) </del> <ins> expected := []api.EnvVar{ {Name: "FOO_BAR_SERVICE_HOST", Value: "machine"}, {Name: "FOO_BAR_SERVICE_PORT", Value: "8080"}, {Name: "FOO_BAR_PORT", Value: "tcp://machine:8080"}, {Name: "FOO_BAR_PORT_8080_TCP", Value: "tcp://machine:8080"}, {Name: "FOO_BAR_PORT_8080_TCP_PROTO", Value: "tcp"}, {Name: "FOO_BAR_PORT_8080_TCP_PORT", Value: "8080"}, {Name: "FOO_BAR_PORT_8080_TCP_ADDR", Value: "machine"}, {Name: "ABC_123_SERVICE_HOST", Value: "machine"}, {Name: "ABC_123_SERVICE_PORT", Value: "8081"}, {Name: "ABC_123_PORT", Value: "udp://machine:8081"}, {Name: "ABC_123_PORT_8081_UDP", Value: "udp://machine:8081"}, {Name: "ABC_123_PORT_8081_UDP_PROTO", Value: "udp"}, {Name: "ABC_123_PORT_8081_UDP_PORT", Value: "8081"}, {Name: "ABC_123_PORT_8081_UDP_ADDR", Value: "machine"}, {Name: "Q_U_U_X_SERVICE_HOST", Value: "machine"}, {Name: "Q_U_U_X_SERVICE_PORT", Value: "8082"}, {Name: "Q_U_U_X_PORT", Value: "tcp://machine:8082"}, {Name: "Q_U_U_X_PORT_8082_TCP", Value: "tcp://machine:8082"}, {Name: "Q_U_U_X_PORT_8082_TCP_PROTO", Value: "tcp"}, {Name: "Q_U_U_X_PORT_8082_TCP_PORT", Value: "8082"}, {Name: "Q_U_U_X_PORT_8082_TCP_ADDR", Value: "machine"}, {Name: "SERVICE_HOST", Value: "machine"}, } if len(vars) != len(expected) { t.Errorf("Expected %d env vars, got: %+v", len(expected), vars) return } for i := range expected { if !reflect.DeepEqual(vars[i], expected[i]) { t.Errorf("expected %#v, got %#v", vars[i], expected[i]) </ins> } } }

We have seen that on VirtualBox in particular kubelet picks first interfaces, which is usually NATed and is of no use. In particular the issue comes up with service IP, which gets mapped to endpoint on the NATed interface.
If this is still a problem, can you explain more - I don't quite understand - and suggest heuristics for better solutions?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: usesEmailVerifiedClaim bool } <del> func validateIssuer(issuer api.Issuer, disallowedIssuers sets.Set[string], fldPath *field.Path) field.ErrorList { </del> <ins> func validateIssuer(issuer api.Issuer, disallowedIssuers sets.Set[string], fldPath *field.Path, structuredAuthnFeatureEnabled bool) field.ErrorList { </ins> var allErrs field.ErrorList allErrs = append(allErrs, validateIssuerURL(issuer.URL, disallowedIssuers, fldPath.Child("url"))...) <del> allErrs = append(allErrs, validateIssuerDiscoveryURL(issuer.URL, issuer.DiscoveryURL, fldPath.Child("discoveryURL"))...) allErrs = append(allErrs, validateAudiences(issuer.Audiences, issuer.AudienceMatchPolicy, fldPath.Child("audiences"), fldPath.Child("audienceMatchPolicy"))...) </del> <ins> allErrs = append(allErrs, validateIssuerDiscoveryURL(issuer.URL, issuer.DiscoveryURL, fldPath.Child("discoveryURL"), structuredAuthnFeatureEnabled)...) allErrs = append(allErrs, validateAudiences(issuer.Audiences, issuer.AudienceMatchPolicy, fldPath.Child("audiences"), fldPath.Child("audienceMatchPolicy"), structuredAuthnFeatureEnabled)...) </ins> allErrs = append(allErrs, validateCertificateAuthority(issuer.CertificateAuthority, fldPath.Child("certificateAuthority"))...) return allErrs

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question: varOpts = append(varOpts, cel.Variable("authorizer", library.AuthorizerType)) } <del> env, err := baseEnv.Extend( </del> <ins> return baseEnv.Extend( </ins> environment.VersionedOptions{ // Feature epoch was actually 1.26, but we artificially set it to 1.0 because these // options should always be present.

When I load a policy definition like this The object is updated with the following status While this error is reported, the policy does seem to be fully functioning This is similar to the documented examples No error reported on a valid spec Create a a policy that uses variables Observe the status of the newly created policy No response <details</details<detailsn/a </details<details</details<detailskind </details<details</details<details</details
/sig api-machinery
/assign Oops. Good find. Thank you. /triage accepted


Question: cmd.Flags().String("serviceaccount", "", "Service account to set in the pod spec") cmd.Flags().String("port", "", i18n.T("The port that this container exposes.  If --expose is true, this is also the port used by the service that is created.")) cmd.Flags().Int("hostport", -1, "The host port mapping for the container port. To demonstrate a single-machine container.") <del> cmd.Flags().StringP("labels", "l", "", "Labels to apply to the pod(s).") </del> <ins> cmd.Flags().StringP("labels", "l", "", "Comma separated labels to apply to the pod(s). Will override previous values.") </ins> cmd.Flags().BoolP("stdin", "i", false, "Keep stdin open on the container(s) in the pod, even if nothing is attached.") cmd.Flags().BoolP("tty", "t", false, "Allocated a TTY for each container in the pod.") cmd.Flags().Bool("attach", false, "If true, wait for the Pod to start running, and then attach to the Pod as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true. With '--restart=Never' the exit code of the container process is returned.")

If you run a command like: $ kubectl run --image=nginx --labels app=web --labels env=prod web It only uses the latest declaration of and drops the earlier options specified. Especially because the --help documentation on does not specify the format, this is impairing the user experience. I looked around a few other CLIs to see how they handle this. curl, for example, uses to additively collect headers (instead of a comma-separated list), therefore each value specified is used. "docker run" command ignores the earlier --name argument when you specify another one. I am wondering if we can easily detect non-additive/overwriting args specified multiple times in an invocation. /kind bug /sig cli
I think this is reasonable for improving cli tool user experience. For different commands, it may different in the options, but for each of them we can add warning messages at least for the end user to realize what had happened in the backend, especially for the ignored staff
/sig cli
why use for flag instead of ? i don't think should overwrite
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: return err } <del> // This means that if a Service transitions away from a nil selector, any // mirrored EndpointSlices will not be cleaned up. #91072 tracks this issue // for this controller along with the Endpoints and EndpointSlice // controllers. </del> <ins> // If a selector is specified, clean up any mirrored slices. </ins> if svc.Spec.Selector != nil { <del> return nil </del> <ins> klog.V(4).Infof("%s/%s Service now has selector, cleaning up any mirrored EndpointSlices", namespace, name) c.endpointSliceTracker.DeleteService(namespace, name) return c.deleteMirroredSlices(namespace, name) </ins> } endpointSlices, err := endpointSlicesMirroredForService(c.endpointSliceLister, namespace, name)

with Selector is created is removed Endpoints and EndpointSlice controllers do not clean up Endpoint(Slice)s mirroring controller starts mirroring the Endpoints resource originally created by the Endpoints controller is back to the Service backing Service change created by mirroring controller still references original IP from step 4 At least one of the following: Endpoints and EndpointSlices to be cleaned up when the Service has the selector removed () mirrored EndpointSlice to be removed when the Service has a selector back See "What Happened" No response Most noticeable in Kubernetes 1.19+ since that's when kube-proxy starts reading from EndpointSlices by default. Any No response No response No response No response
/sig network /triage accepted /priority important-soon /cc
/assign


Question: } // TODO: Find a uniform way to deal with systemctl/initctl/service operations. #34494 <del> func restartKubelet() { </del> <ins> func findRunningKubletServiceName() string { </ins> stdout, err := exec.Command("sudo", "systemctl", "list-units", "kubelet*", "--state=running").CombinedOutput() framework.ExpectNoError(err) regex := regexp.MustCompile("(kubelet-w+)") matches := regex.FindStringSubmatch(string(stdout)) <del> framework.ExpectNotEqual(len(matches), 0) kube := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kube) stdout, err = exec.Command("sudo", "systemctl", "restart", kube).CombinedOutput() </del> <ins> framework.ExpectNotEqual(len(matches), 0, "Found more than one kubelet service running: %q", stdout) kubeletServiceName := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kubeletServiceName) return kubeletServiceName } func restartKubelet() { kubeletServiceName := findRunningKubletServiceName() stdout, err := exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } <ins> // stopKubelet will kill the running kubelet, and returns a func that will restart the process again func stopKubelet() func() { kubeletServiceName := findRunningKubletServiceName() stdout, err := exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to stop kubelet with systemctl: %v, %v", err, stdout) return func() { stdout, err := exec.Command("sudo", "systemctl", "start", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } } </ins> func toCgroupFsName(cgroupName cm.CgroupName) string { if framework.TestContext.KubeletConfig.CgroupDriver == "systemd" { return cgroupName.ToSystemd()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files.

I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?


Question: "//vendor/k8s.io/utils/nsenter:go_default_library", ], "@io_bazel_rules_go//go/platform:windows": [ <ins> "//vendor/k8s.io/utils/keymutex:go_default_library", </ins> "//vendor/k8s.io/utils/nsenter:go_default_library", "//vendor/k8s.io/utils/path:go_default_library", ],

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am using Windows containers (Server Core ltsc 2019). These all have volumes mounted to three different Azure file shares under a single storage account. Multiple instances of these containers run on each node. On deploying the containers I get the following errors. Sometimes this does manage to resolve itself. MountVolume.SetUp failed for volume "log-share" : Remove-SmbGlobalMapping failed: exit status 1, output: "Remove-SmbGlobalMapping : No MSFTSmbGlobalMapping objects found with property 'RemotePath' equal to rn'storageaccounturllogs'. Verify the value of the property and retry.rnAt line:1 char:1rn+ Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Forcern+ ~rn + CategoryInfo : ObjectNotFound: (storageaccounturllogs:String) [Remove-SmbGlobalMapping], Ci rn mJobExceptionrn + FullyQualifiedErrorId : CmdletizationQueryNotFoundRemotePath,Remove-SmbGlobalMappingrn rn" MountVolume.SetUp failed for volume "config-share" : New-SmbGlobalMapping failed: exit status 1, output: "New-SmbGlobalMapping : Generic failure rnAt line:1 char:190rn+ ... ser, Env:smbremotepath -Cred ...rn+ ~~~rn + CategoryInfo : NotSpecified: (MSFTSmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa rn pping], CimExceptionrn + FullyQualifiedErrorId : HRESULT 0x80041001,New-SmbGlobalMappingrn rn" What you expected to happen: For the volumes to be mounted. This all worked ok in k8s 1.12.1 using Windows 1803. I have now upgraded to 1.13.4 and Windows ltsc2019.

How to reproduce it (as minimally and precisely as possible): Use aks-engine to create a 1.13.4 cluster with Windows nodes Create a storage account Create multiple file shares under the account. Add access credentials to the storage account as a secret on k8s Create a windows container with folders that will be used to mount the Azure storage account file shares. Deploy to k8s defining volume mounts and volumes in the container section of the deployment e.g. volumeMounts: name: folder1-share mountPath: "c:appfolder1" name: folder2-share mountPath: "c:appfolder2" volumes: name: folder1-share azureFile: secretName: azure-file-share-cred shareName: share1 readOnly: false name: folder2-share azureFile: secretName: azure-file-share-cred shareName: share2 readOnly: true Anything else we need to know?: This maybe related to the fix for Environment: Kubernetes version (use ): 1.13.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Windows 2019 Kernel (e.g. ): 10.0..316 Install tools: aks-engine 0.13.2 Others:
/sig Azure
/sig windows
/assign
hi in your logs, the remotepath is , so is the share name? also, could you log on to that agent node, and provide , there are logs like
I think this is a race condition issue, let me file a PR to fix this, thanks for reporting


Question: for name, indexFunc := range c.indexers { indexValues, err := indexFunc(newObj) if err != nil { <del> return err </del> <ins> panic(fmt.Errorf("unable to calculate an index entry for key %q on index %q: %v", key, name, err)) </ins> } index := c.indices[name] if index == nil {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): k8s1.3.5. still exist in 1.6 What happened: when exec threadSafeMap.Index func, the returned items contain nil object. That is to say, index is in threadSafeMap but the corresponding has already been deleted. What you expected to happen: when object is deleted from threadSafeMap, all its index must be deleted. How to reproduce it (as minimally and precisely as possible): Suppose we init threadSafeMap with two indexers: indexFunc1 and indexFunc2. Obviously, indexFunc may return error if the object has something wrong or some other reasons. when we add an object, will first add the object to , then compute the index and add the index into threadSafe.indices. when we delete the object, will first delete the index from indices, then delete it from Suppose IndexFunc1(object) return error and indexFunc2(object) works well. I think this circumstance is absolutely possible to happen. Also we know that the order to exec indexFunc is not decided. Feature of iterating map. When we add object, suppose first exec indexFunc2, and then exec indexFunc1. So the index of indexFunc2 is to indices. But the index of indexFunc1 is not , because indexFunc1(obj) return err. Then we delete the object, suppose first exec indexFunc1. It will return error and IndexFunc2 has no chance to run. So the corresponding index indexFunc2(obj) can't be removed. But then the object is removed from items. This problem may result in subtle problems. For example, when we exec threadSafeMap.Index( ) func the returned list may include nil object. Anything else we need to know The reason is that index is still is threadSafeMap but the corresponding object has already been removed. We must clear all corresponding indexs when delete object. And the reason why index is not deleted is that when for-range map, the order is not decided.
Who can help me? Do i miss something important or misunderstand it ?
, can you share a unit test to show it? I'd like to help on it.
Thank you for your help. I think i have decribed the condition resulting in the bug indetail.

Be simple, because the order of exec indexFunc is random when update indices, and some function don't care the error returned, the obj is deleted but corresponding indices is still in it. NOTE: the obj is deleted but corresponding indices is still in it So when we run index func, the returned list may contain nil object. Because when get obj from threadsafemap with indice, not care whether the obj exist in this map. Then nil object .
. I am sure this problem exists. And i think we can find a better way to deal with it.
/sig api-machinery threadSafeMap is in client-go
/kind bug
Opened a PR with a fix.
In my understanding, should be deterministic, could you elaborate more how your IndexFunc implemented? If an object can not be called on this , then the question is, how could it exist in at the first beginning?
threadSafeStore should tolerate some bad conditions and deal with it gracefully, instead generating strange and confusing phenomenon. ThreadSafeStore can contain mutiple index funcs. If one of them is not implemented very well, or one of objects we pass to it has something wrong, STRANGE AND CONFUSING phenomenon happens, see what I said above.
That's the point I would like to know more, the call on an object should succeed or fail unconditionally(deterministic).
wechat
I also think index function must be a pure function. See my comment on the pr


Question: for arch in "${KUBE_SERVER_PLATFORMS[@]##*/}"; do for binary in "${binaries[@]}"; do <del> local docker_target="${KUBE_DOCKER_REGISTRY}/${binary}-${arch}:${KUBE_DOCKER_IMAGE_TAG}" kube::log::status "Pushing ${binary} to ${docker_target}" "${docker_push_cmd[@]}" push "${docker_target}" </del> <ins> # TODO(IBM): Enable hyperkube builds for ppc64le again if [[ ${binary} != "hyperkube" || ${arch} != "ppc64le" ]]; then local docker_target="${KUBE_DOCKER_REGISTRY}/${binary}-${arch}:${KUBE_DOCKER_IMAGE_TAG}" kube::log::status "Pushing ${binary} to ${docker_target}" "${docker_push_cmd[@]}" push "${docker_target}" </ins> <del> # If we have a amd64 docker image. Tag it without -amd64 also and push it for compatibility with earlier versions if [[ ${arch} == "amd64" ]]; then local legacy_docker_target="${KUBE_DOCKER_REGISTRY}/${binary}:${KUBE_DOCKER_IMAGE_TAG}" </del> <ins> # If we have a amd64 docker image. Tag it without -amd64 also and push it for compatibility with earlier versions if [[ ${arch} == "amd64" ]]; then local legacy_docker_target="${KUBE_DOCKER_REGISTRY}/${binary}:${KUBE_DOCKER_IMAGE_TAG}" </ins> <del> "${DOCKER[@]}" tag -f "${docker_target}" "${legacy_docker_target}" 2>/dev/null </del> <ins> "${DOCKER[@]}" tag -f "${docker_target}" "${legacy_docker_target}" 2>/dev/null </ins> <del> kube::log::status "Pushing ${binary} to ${legacy_docker_target}" "${docker_push_cmd[@]}" push "${legacy_docker_target}" </del> <ins> kube::log::status "Pushing ${binary} to ${legacy_docker_target}" "${docker_push_cmd[@]}" push "${legacy_docker_target}" fi </ins> fi done done

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: CSIMigrationOpenStack:          {Default: false, PreRelease: featuregate.Beta}, // Off by default (requires OpenStack Cinder CSI driver) CSIMigrationOpenStackComplete:  {Default: false, PreRelease: featuregate.Alpha}, VolumeSubpath:                  {Default: true, PreRelease: featuregate.GA}, <del> ConfigurableFSGroupPolicy:      {Default: false, PreRelease: featuregate.Alpha}, </del> <ins> ConfigurableFSGroupPolicy:      {Default: true, PreRelease: featuregate.Beta}, </ins> BalanceAttachedNodeVolumes:     {Default: false, PreRelease: featuregate.Alpha}, CSIBlockVolume:                 {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 CSIInlineVolume:                {Default: true, PreRelease: featuregate.Beta},

We need to add e2e that verifies volume permissions in pod with different values of fsGroupChangePolicy. /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: CSIMigrationOpenStack:          {Default: false, PreRelease: featuregate.Beta}, // Off by default (requires OpenStack Cinder CSI driver) CSIMigrationOpenStackComplete:  {Default: false, PreRelease: featuregate.Alpha}, VolumeSubpath:                  {Default: true, PreRelease: featuregate.GA}, <del> ConfigurableFSGroupPolicy:      {Default: false, PreRelease: featuregate.Alpha}, </del> <ins> ConfigurableFSGroupPolicy:      {Default: true, PreRelease: featuregate.Beta}, </ins> BalanceAttachedNodeVolumes:     {Default: false, PreRelease: featuregate.Alpha}, CSIBlockVolume:                 {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 CSIInlineVolume:                {Default: true, PreRelease: featuregate.Beta},

We need to move this field to beta in 1.20. /sig storage
/triage accepted
According to , is working on a better design on this.
Nope. The current design is the design we are going to use to handle fsgroup. we discussed this at length in sig-storage call.
Is it okay if I make this change?


Question: ![Services overview diagram](services_overview.png) <ins> By default, the choice of backend is random.  Client-IP-based session affinity can be selected by setting `service.spec.sessionAffinity` to  `"ClientIP"`. </ins> ### Why not use round-robin DNS? A question that pops up every now and then is why we do all this stuff with

Within a Kubernetes cluster each node runs a service proxy that will round robin load balance requests to the various pods within the cluster. Currently there is no mechanism to maintain session affinity for applications that require sticky sessions. To enable session affinity we need to modify the load balancer resides within each node's service proxy to direct requests to specific endpoints based on a client IP address (future enhancement would be to support cookie based session ID). From a high level, I propose the following changes to enable support for session affinity. SessionAffinity to ServiceSpec in (I welcome better object names): a map, serviceDtlMap to the LoadBalanacerRR strcut in The serviceDtlMap will be a map whose key is the service name and value will be a new type called serviceDetail. The serviceDetail struct will hold information about the session affinity settings (is it enabled and the type of session affinity - cookie or client IP) along with a map of ssessionAffinityMap. The sessionAffinityMap will have a key that is either the clientIP or the cookie and the value will a new type called sessionAffinityDetail. The sessionAffinityDetail struct will hold the client IP address, the session cookie, the endpoint url and the last used date and time (last used DTTM will be used for cleaning up old records). the OnUpdate method in to also update the LoadBalancerRR.sessionAffinityMap when new services or endpoints are or removed. LoadBalancerRR NextEndpoint method () to first check the session affinity map for an endpoint based on either the cookie or the client ip address (based on preference in step 1 above). If an endpoint exists for this request, then return that endpoint - otherwise continue with the existing flow of selecting next endpoint and then add this endpoint to the sessionAffnityMap. a background task to clean up stale session affinity records based on some life time setting (say 30 or 60 minutes of inactivity). Note that the external load balancer will also require some form of session affinity, otherwise it could send a client request to node who's service proxy does not have a record for the client in it's session affinity map.
Is this implemented now?
It is implemented, sorry for leaving it open. I had planned to add some documentation (was thinking of adding to guestbook example) while on vacation, but that didn't happen.

I'll close this one and create a separate ticket.
Does this have a doc?
Set to "ClientIP" , Harry Zhang wrote:
We try to use ClientIP session affinity with NodePort type (as opposed to LoadBalancer type) on AWS (for cookie-based HTTP traffic). We have manually created an Amazon load-balancer. ( The AWS load-balancer may use cookie-based session affinity or any other session affinity ). Is ClientIP session affinity (using NodePort) supported also on AWS? Was this implemented & tested on AWS? It seems that AWS load-balancer is exposing its own IP address to Kubernetes - so ClientIP affinity is using only a single pod on each node (the pod that is related to the AWS load-balancer IP address)? Is that so? If so, how can we configure Amazon load-balancer to expose the source client IP? Is that possible? Otherwise, it seems that ClientIP session affinity (using NodePort) is not applicable on AWS....
ELB terminates the client TCP connection and opens a new one. Therefore, the clientIP affinity mode does not work properly with ELB. I don't have a better answer, unfortunately. are there any clever workarounds?
one way to achieve this is enabling proxy protocol on ELB and using an intermediary nginx container to pass the real IP to your service (maybe this is now supported directly in kube-proxy or ingress. Is it ?) I had written a blog post on this topic -
PROXY protocol is not directly supported - we'd have to peek at the TCP stream to do that.
Turns out some of the info I had given was incorrect. Sorry about that. You can check the full discussion here -
Can you take your post down if it's not correct? People (such as myself) are finding your post and thinking it's the proper solution, only to have problems and dig down to find these issues. By the way, really appreciate you being the trailblazer and investigating this matter- we have this need as well.
For future reference: kube-proxy is L4. If your source IP is the same, you can't distinguish clients. You need a L7 proxy to read those details. Check Ingress: This uses cookies to identify your user and skips the L4 kube-proxy invoking the pod directly.


Question: } var map_Binding = map[string]string{ <del> "":         "Binding ties one object to another. For example, a pod is bound to a node by a scheduler.", </del> <ins> "":         "Binding ties one object to another; for example, a pod is bound to a node by a scheduler. Deprecated in 1.7, please use the bindings subresource of pods instead.", </ins> "metadata": "Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata", "target":   "The target object that you want to bind to the standard object.", }

Forked from We need to update the scheduler (ours and mesos) to use pods/binding instead of /binding and then we can delete /binding.
cc
This is about the top-level resource, not the subresource.
This is where mesos is using binding as top level resource: We need to change that to Resource("pods").Subresource("bindings")
/cc
If we turn this off in 1.2, will that be a problem for you?
Did we announce we were going to remove this API? Don't we need to have a process for taking it away?
Yes it would be problematic unless our scheduler was updated to use whatever was replacing it. AFAICT even the core scheduler is still using /bindings: , Clayton Coleman wrote:
Kicking this can down the road.
We should mark this deprecated.
According to , kube-mesos refactor will use HTTPExtender, so it's a not big issue to kube-mesos. But for k8s, I think scheduler is using API now ().
/assign
/close I'm going to close this; if any comments, please let me know.
We need to actually mark the endpoint deprecated so we can remove it.
OK, got that; need to deprecate it for other user. I think k8s can handle it to use .
, can you help to remove ? it's not related with Mesos anymore. I'll create a PR to deprecate .
Reopening since that PR just a deprecation warning. We still need to remove the API.
wow, agree. , would you help to remove milestone? We will not remove API in 1.7 or 1.8; but I'll trace that.
Done
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
18 still have.
21 still have the resource. Is anyone already working on it or is there a reason why the binding resource still get served?

not entirely sure if i'm right, but from my current assumption it seems that there isn't an implementation for the for pods (xref: ) and we still use the generic resource for pods (). The resource is deprecated since 1.7 but i don't see any further progress :thinking: () dump of an :
what is the progress with this issue? Is it a backlog item that is tracked by sig scheduling?
I'm not aware of this issue. We will put it into our sig meeting to discuss.
we went through this in today's scheduling meeting. We're ok with shifting v1.Binding to pod's subresource. Before acting on it, there are several factors we need to consider: impact to external integrators - either 3rd-party scheduler or scheduler plugins (using extension point). Follow the conventional API deprecation/removal guide/process to ensure a smooth migration. overall volume of efforts. In addition to scheduling efforts, are there any API Server side efforts that need to be tweaked as well? we need to get +1 from
Is there a summary of why moving this API is necessary given that is has been around for quite sometime now? Is it related to RBAC?
/binding is a v1 API, and as such, deletion is not permitted. It can be marked deprecated, and users pointed to pods/binding by documentation as the preferred method, but not removed.
It's more to pay the technical debt. Given the action "binding" is performed against pods, composing a pod/binding instead of v1.Binding makes more sense. Thanks for clarification we cannot remove v1 APIs.
ok, thanks for the clarification, so this is not enabling a new use case, only a cleanup.
Right, and due to the stable API compatibility guarantees, I don't think it's a cleanup we can do. /close
Closing this issue. <detailsIn response to : so this is not enabling a new use case, only a cleanup. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry for the late comment on this but i was AFK for a longer time. I'm not sure if we missed something. Currently there is only the implementation.

via the the API endpoint exists but it refers to the only existing kind (could be checked via ): As there isn't a seperate implementation for IMO we should remove the deprecation warning as we don't have an alternative implementation for that in place. / WDYT?


Question: ObjectMeta: api.ObjectMeta{ Name:      name, Namespace: ns, <ins> Annotations: map[string]string{ "pod.alpha.kubernetes.io/initialized": "false", }, </ins> }, Spec: apps.StatefulSetSpec{ Selector: &unversioned.LabelSelector{

PetSets currently support an annotation in their PodTemplateSpec field. From the user documentation: Debugging: If one want to bring up the pets in my petset one after the other, while pausing after each step, confirming that the state is as expected, one may use this functionality. For example, a petset where each pet is a replica of a distributed database; we set the to in the petset template initially. This leads to the following progression: PetSetController creates pet-0 Manually intervene and check that it has initialized as expected. Set annotation to true (equivalent to marking that pet as healthy) PetSetController creates pet-1 Manually intervene and check that it has initialized as expected. ... Blocking the PetSet controller: We may want to ensure that the petset controller is not performing scale operations when a different user-initiated action is in progress (for example, where pets are leaving due to a kubectl drain). This may be used as a mechanism to orphan a pod from a petset controller (for a closer look), while ensuring that a replacement is not created. The use case 1 above can be realized using and is probably easier to do that way rather than use the annotation. I don't see a reason to have that case supported when we can have users scale up a PetSet gradually and manually in an identical manner. It may have been of more importance before v1.4 when the scaler & reaper were not implemented. Due to the second use-case, if it would make sense to continue to have this ability, I propose that we move it from the PodTemplateSpec to the PetSetSpec, where it will eventually reside. This seems to me, like the logical place for such a debug hook to reside. In order to run/pause the scaling operations of the PetSet controller, we could have annotations along the lines of with the annotation defaulting to when absent. /cc
Should we just support Pause on PetSet (for part of these cases)?
Do users have to catch the roll out at the right time in order for this to work? For example if I want to pause between nodes 2 and 3 coming up, do I need to be precise in my timing when editing the resource? How does this look from the users perspective?
Note that Pause allows scaling in Deployments, it just blocks upgrades.

That being said, I think Pause will be useful for PetSets in general. Not sure though if we can use it somehow for this specific issue.
I like the debugging hook for PetSets. If it's optional I don't see any reason to get rid of it.
pauses petset controller if necessary and inspects the state of the system_ With the previous flow, the petset controller was paused after each pet's creation automatically, and here it would be an extra step to pause the controller it if necessary. I don't know how important it is though, but to me it seems easier to understand than a per-pod annotation.
and I had several discussions with the lifecycle of what would be useful. There are a bunch of use cases that overlap: not a primary concern for most users but helpful (prevent changes) - very important if you are triaging failures. Technically scaling is itself a user action, so if you don't want your pet set autoscaled while triaging failures you need to take action on the HPA like deployments. Since PetSets and ReplicaSets are nothing alike w.r.t. scaling I'm not terribly concerned that pause means slightly different things if for end users it Does The Right Thing control by a third party controller - this is the babysitter controller wanting to reuse the petset controller for core stuff but still have the hook angle. Not thought through enough to design around.
Note that Pause does not prevent scaling in Deployments, only upgrades , Clayton Coleman wrote:
Just to summarize: The current behavior with the annotation It is set on a per-pet basis. Setting it to false on any single pod blocks all operations on the petset; scaling, creation of new pets, etc. All pets must contain the annotation, and have its value set to true in order to make progress. By default, it is set to false and makes the petset controller pause after creating one pet. The smallest change we can make for beta: Rename it to (or some better name) Make it optional (with the absence of the annotation meaning that the petset can scale) Surface the reason better when we block on scaling because of the annotation. I still think there is value in moving it to be an annotation on the PetSet resource, rather than having it be in each pod, which is confusing to users. wdyt?
Putting it on the pod template is clever, but surprising.

How many examples are dependent on it? , Anirudh Ramanathan < :
Last I checked, all the examples had set to true. We should just get rid of it from all examples in my opinion. If we make the default (or the absence of any such annotation) to mean that the petset controller can make progress, we can safely do that. was going to do a pass over the examples after the PetSet renaming. This could be an additional step. Does that sound ok?
If it's not found, we assume you want to pause. You're proposing that if it's not found, we default to "forge ahead". SG. It should only be used in debugging/testing today. Basically if you don't know how to write your , you'd create a petset with it set, kubectl exec and write it out, scale the petset by 1, kubectl exec ..., and when you have a working cluster, put what you did in an init container and create the petset without it. This is the absolute min. You're also proposing that we move it to the petset. This makes sense overall, but I initially didn't do that because I didn't know how we would end up doing pause.
We could leave it there as "alpha" and then remove it in 1.6 once we have more time to talk about how upgrades and debugging and pause should work? , Prashanth B wrote:
Yeah the minimum work is to flip the behavior of the controller when the annotation is not found:
Agree on flipping behavior. , Prashanth B wrote:
Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from as well, since it doesn't make as much sense anymore. Probably ?
or healthy rather.
Flipping default behavior (when not present). Or did I misunderstand beeps proposal? On Oct 27, 2016, at 7:11 PM, Anirudh Ramanathan wrote: Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from initialized as well, since it doesn't make as much sense anymore. Probably unhealthy?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Renaming it to beta is effectively moving it to beta, so I think given that we're still confused about how this ties in to the big picture, that might be premature. If it's either move to beta or drop, I'd probably vote for drop, it's a debug hook after all. Renaming it to something else but keeping in beta makes sense.
sorry, keeping in alpha
whats going on with this tag? it's used in the docs here should i use it going forward?
It's alpha and may be removed at any time. On Nov 2, 2016, at 3:11 PM, Zachary Adam Kaplan wrote: whats going on with this tag? it's used in the docs here should i use it going forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .


Question: return nil, nil } isInternal := requiresInternalLoadBalancer(service) <del> lbFrontendIPConfigName := az.getFrontendIPConfigName(service, subnet(service)) </del> <ins> lbFrontendIPConfigName := az.getFrontendIPConfigName(service) </ins> serviceName := getServiceName(service) for _, ipConfiguration := range *lb.FrontendIPConfigurations { if lbFrontendIPConfigName == *ipConfiguration.Name {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: func TestRunApplyPrintsValidObjectList(t *testing.T) { cmdtesting.InitTestErrorHandler(t) <del> cmBytes := readConfigMapList(t, filenameCM) </del> <ins> configMapList := readConfigMapList(t, filenameCM) </ins> pathCM := "/namespaces/test/configmaps" tf := cmdtesting.NewTestFactory().WithNamespace("test")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: } // Marshal to disk <del> if err := clientcmd.WriteToFile(kubeconfigData, kubeconfigPath); err != nil { return err } return nil </del> <ins> return clientcmd.WriteToFile(kubeconfigData, kubeconfigPath) </ins> } func loadRESTClientConfig(kubeconfig string) (*restclient.Config, error) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: reports following errors- What you expected to happen: This is the required fix reported by PR processing. How to reproduce it (as minimally and precisely as possible): run from k8s source. Anything else we need to know?: Detailed list of ERRORS reported by Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig-testing-bugs
It looks like you're building with go1.9.x… master is building with go1.10 now, so gofmt expects to run with that version
/close +1 to what said