Question: // Config holds the information needed to build connect to remote kubernetes clusters as a given user type Config struct { <del> api.TypeMeta `json:",inline"` </del> <ins> // Legacy field from pkg/api/types.go TypeMeta. // TODO(jlowdermilk): remove this after eliminating downstream dependencies. Kind string `json:"kind,omitempty"` // Version of the schema for this config object. APIVersion string `json:"apiVersion,omitempty"` </ins> // Preferences holds general information to be use for cli interactions Preferences Preferences `json:"preferences"` // Clusters is a map of referencable names to cluster configs

has a dependency on pkg/api for , which it embeds in . Is it doing that to take advantage of validation code, or can Config just declare and directly and avoid the dependency?
A quick try (just declaring a in the package) seems to work fine with . I only included those out of habit.
Out of curiosity, is this just for a proper separate of packages or is there some functional issue as well?
mentioned encountering an issue with tests, but it's mostly just for proper separation of packages.
Related: Remove json tags from
See also
Not done:


Question: exit 1 fi <ins> do_backup_clean </ins> # use an array to record name and ip declare -A mm

I used the ubuntu-cluster setup scripts to build a mini testing environment. The first time I inputted a wrong config and when I reconfigured the environment by running the scripts again an error occured. I looked into the code and found the scripts hadn't deal with old configs. And of kube-proxy configs are not replaced.
Looks like this was fixed by .


Question: sync.Mutex input   chan watchCacheEvent result  chan watch.Event <ins> done    chan struct{} </ins> filter  watchFilterFunc stopped bool forget  func(bool)

We noticed many goroutines stalled at in a large cluster (~200 nodes, ~1100 pods). It's because if the WatchServer timeout when the channel is full, will be blocked on the write to the even though is , because WatchServer stops consuming the channel after the timeout.
Great finding - thanks for debugging


Question: --query Reservations[].Instances[].NetworkInterfaces[0].Association.PublicIp } <ins> function get_instance_private_ip { local instance_id=$1 $AWS_CMD --output text describe-instances  --instance-ids ${instance_id}  --query Reservations[].Instances[].NetworkInterfaces[0].PrivateIpAddress } </ins> # Gets a security group id, by name ($1) function get_security_group_id { local name=$1

When you run the "Cluster / " error below line 182: getinstanceprivate_ip: command not found i add the function in cluster/aws/util.s
, André Ribeiro de Miranda < :
any chance you can send a PR? Thanks! --brendan
Actually, let me send a PR. This should just be reversing a delete I made (in error).
if you do want to put together a PR we can of course use that instead, but I deleted this by mistake so I have created a PR to fix my mistake.
Believe also need to change the function check-minion. To make a "ssh forward" by the master. More has the following comment which makes check-minion in the file " 54" I would make this change the "ssh foward" did not do more because of that comment above Thanks
When I am running: export KUBERNETESPROVIDER=aws; wget -q -O - | bash with ENABLENODEPUBLICIP all works fine and the cluster is up, though when running with KUBEENABLENODEPUBLICIP = 0, the cluster won't go up. why the public ip's are needed for the nodes? isn't the VPC good enough? Thanks.


Question: } } } <del> expectIngress := false if svcType == v1.ServiceTypeLoadBalancer { expectIngress = true } hasIngress := len(svc.Status.LoadBalancer.Ingress) != 0 if hasIngress != expectIngress { return nil, fmt.Errorf("unexpected number of Status.LoadBalancer.Ingress (%d) for service", len(svc.Status.LoadBalancer.Ingress)) } if hasIngress { for i, ing := range svc.Status.LoadBalancer.Ingress { if ing.IP == "" && ing.Hostname == "" { return nil, fmt.Errorf("unexpected Status.LoadBalancer.Ingress[%d] for service: %#v", i, ing) } </del> <ins> if svcType != v1.ServiceTypeLoadBalancer { if len(svc.Status.LoadBalancer.Ingress) != 0 { return nil, fmt.Errorf("unexpected Status.LoadBalancer.Ingress on non-LoadBalancer service") </ins> } }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:

Which jobs are failing: Which test(s) are failing: Since when has it been failing: 18 Oct 13:17 PDT Testgrid link: https://k8s- Reason for failure: check
/sig network /cc /cc
/priority critical-urgent /milestone v1.17
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I believe this is caused by this merge cc.
Most of this is caused by and should be fixed by . Is there any way to run these tests against that PR? /remove-triage unresolved
The job is still failing. Can you check?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage unresolved
OK, fixed some of the tests, got the rest to the point where they failed differently...
Thanks for the update. In testgrid we still see the same error. We will keep observing the next runs.
- still failing. This time the error is We probably need to clean up sanityChecks a bit more - this seems to be a similar issue.
yeah, I'll temporarily un- the tests in my next PR so I can make sure they're actually fixed this time
After the last fix, it seemed to finally recover.


Question: plugin, ok := w.getPlugin(event.Name) if !ok { <del> return fmt.Errorf("could not find plugin for deleted file %s", event.Name) </del> <ins> klog.V(5).Infof("could not find plugin for deleted file %s", event.Name) return nil </ins> } // You should not get a Deregister call while registering a plugin

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: options = append(options, "ro") } <del> l := len(b.hosts.Subsets) </del> <ins> p := path.Join(b.glusterfs.plugin.host.GetPluginDir(glusterfsPluginName), b.glusterfs.volName) if err := os.MkdirAll(p, 0750); err != nil { return err } log := path.Join(p, "glusterfs.log") options = append(options, "log-file="+log) addr := make(map[string]struct{}) for _, s := range b.hosts.Subsets { for _, a := range s.Addresses { addr[a.IP] = struct{}{} } } </ins> // Avoid mount storm, pick a host randomly. <del> start := rand.Int() % l </del> // Iterate all hosts until mount succeeds. <del> for i := start; i < start+l; i++ { hostIP := b.hosts.Subsets[i%l].Addresses[0].IP </del> <ins> for hostIP := range addr { </ins> errs = b.mounter.Mount(hostIP+":"+b.path, dir, "glusterfs", options) if errs == nil { return nil

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-aks-engine-azure Which test(s) are failing: Since when has it been failing: Testgrid link: No Testgrid, but job history: Reason for failure: Anything else we need to know: ref:
/assign cc: /milestone v1.16 /priority important-soon
/sig azure
cc
Related with
Update: has been merged, but a new release with this change is not available yet. Pending on aks-engine's next release.
/sig cloud-provider
v1.16.0-alpha.1 is available in
Will submit a PR after testing this in our own prow first.


Question: - name: kube-dns-config mountPath: /etc/k8s/dns/dnsmasq-nanny - name: sidecar <del> image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.2 </del> <ins> image: gcr.io/google_containers/k8s-dns-sidecar-amd64:1.14.3 </ins> livenessProbe: httpGet: path: /metrics

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: } } <ins> func TestValidateObjectMetaUpdatePreventsDeletionFieldMutation(t *testing.T) { now := unversioned.NewTime(time.Unix(1000, 0).UTC()) later := unversioned.NewTime(time.Unix(2000, 0).UTC()) gracePeriodShort := int64(30) gracePeriodLong := int64(40) testcases := map[string]struct { Old          api.ObjectMeta New          api.ObjectMeta ExpectedNew  api.ObjectMeta ExpectedErrs []string }{ "valid without deletion fields": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1"}, ExpectedErrs: []string{}, }, "valid with deletion fields": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now, DeletionGracePeriodSeconds: &gracePeriodShort}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now, DeletionGracePeriodSeconds: &gracePeriodShort}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now, DeletionGracePeriodSeconds: &gracePeriodShort}, ExpectedErrs: []string{}, }, "invalid set deletionTimestamp": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, ExpectedErrs: []string{"field.deletionTimestamp: Invalid value: "1970-01-01T00:16:40Z": field is immutable; may only be changed via deletion"}, }, "invalid clear deletionTimestamp": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, ExpectedErrs: []string{}, // no errors, validation copies the old value }, "invalid change deletionTimestamp": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &later}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &now}, ExpectedErrs: []string{}, // no errors, validation copies the old value }, "invalid set deletionGracePeriodSeconds": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodShort}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodShort}, ExpectedErrs: []string{"field.deletionGracePeriodSeconds: Invalid value: 30: field is immutable; may only be changed via deletion"}, }, "invalid clear deletionGracePeriodSeconds": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodShort}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1"}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodShort}, ExpectedErrs: []string{}, // no errors, validation copies the old value }, "invalid change deletionGracePeriodSeconds": { Old:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodShort}, New:          api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodLong}, ExpectedNew:  api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionGracePeriodSeconds: &gracePeriodLong}, ExpectedErrs: []string{"field.deletionGracePeriodSeconds: Invalid value: 40: field is immutable; may only be changed via deletion"}, }, } for k, tc := range testcases { errs := ValidateObjectMetaUpdate(&tc.New, &tc.Old, field.NewPath("field")) if len(errs) != len(tc.ExpectedErrs) { t.Logf("%s: Expected: %#v", k, tc.ExpectedErrs) t.Logf("%s: Got: %#v", k, errs) t.Errorf("%s: expected %d errors, got %d", k, len(tc.ExpectedErrs), len(errs)) continue } for i := range errs { if errs[i].Error() != tc.ExpectedErrs[i] { t.Errorf("%s: error #%d: expected %q, got %q", k, i, tc.ExpectedErrs[i], errs[i].Error()) } } if !reflect.DeepEqual(tc.New, tc.ExpectedNew) { t.Errorf("%s: Expected after validation:n%#vngotn%#v", k, tc.ExpectedNew, tc.New) } } } </ins> // Ensure trailing slash is allowed in generate name func TestValidateObjectMetaTrimsTrailingSlash(t *testing.T) { errs := ValidateObjectMeta(

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: // SwaggerSchemaInterface has a method to retrieve the swagger schema. Used in // client.Interface type SwaggerSchemaInterface interface { <del> SwaggerSchema(version string) (*swagger.ApiDeclaration, error) </del> <ins> SwaggerSchema(groupVersion string) (*swagger.ApiDeclaration, error) </ins> } // SwaggerSchema retrieves and parses the swagger API schema the server supports. <del> func (c *Client) SwaggerSchema(version string) (*swagger.ApiDeclaration, error) { if version == "" { version = latest.GroupOrDie("").Version </del> <ins> func (c *Client) SwaggerSchema(groupVersion string) (*swagger.ApiDeclaration, error) { if groupVersion == "" { return nil, fmt.Errorf("groupVersion cannot be empty") </ins> } <del> vers, err := c.ServerAPIVersions() </del> <ins> groupList, err := c.Discovery().ServerGroups() </ins> if err != nil { return nil, err } <ins> groupVersions := extractGroupVersions(groupList) </ins> // This check also takes care the case that kubectl is newer than the running endpoint <del> if stringDoesntExistIn(version, vers.Versions) { return nil, fmt.Errorf("API version: %s is not supported by the server. Use one of: %v", version, vers.Versions) </del> <ins> if stringDoesntExistIn(groupVersion, groupVersions) { return nil, fmt.Errorf("API version: %s is not supported by the server. Use one of: %v", groupVersion, groupVersions) } var path string if groupVersion == "v1" { path = "/swaggerapi/api/" + groupVersion } else { path = "/swaggerapi/apis/" + groupVersion </ins> } <del> body, err := c.Get().AbsPath("/swaggerapi/api/" + version).Do().Raw() </del> <ins> body, err := c.Get().AbsPath(path).Do().Raw() </ins> if err != nil { return nil, err }

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


Question: - selectorRegexp: k8s[.]io/kubernetes/pkg allowedPrefixes: - k8s.io/kubernetes/pkg/api/legacyscheme <ins> - k8s.io/kubernetes/pkg/apis/admission - k8s.io/kubernetes/pkg/apis/admission/install - k8s.io/kubernetes/pkg/apis/admission/v1 - k8s.io/kubernetes/pkg/apis/admission/v1beta1 - k8s.io/kubernetes/pkg/apis/admissionregistration - k8s.io/kubernetes/pkg/apis/admissionregistration/install - k8s.io/kubernetes/pkg/apis/admissionregistration/v1 - k8s.io/kubernetes/pkg/apis/admissionregistration/v1beta1 - k8s.io/kubernetes/pkg/apis/apiserverinternal - k8s.io/kubernetes/pkg/apis/apiserverinternal/fuzzer - k8s.io/kubernetes/pkg/apis/apiserverinternal/install - k8s.io/kubernetes/pkg/apis/apiserverinternal/v1alpha1 </ins> - k8s.io/kubernetes/pkg/apis/apps - k8s.io/kubernetes/pkg/apis/apps/install - k8s.io/kubernetes/pkg/apis/apps/v1

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: var ns string var c clientset.Interface var aggrclient *aggregatorclient.Clientset <ins> // BeforeEachs run in LIFO order, AfterEachs run in FIFO order. // We want cleanTest to happen before the namespace cleanup AfterEach // inserted by NewDefaultFramework, so we put this AfterEach in front // of NewDefaultFramework. AfterEach(func() { cleanTest(c, aggrclient, ns) }) </ins> f := framework.NewDefaultFramework("aggregator") <ins> // We want namespace initialization BeforeEach inserted by // NewDefaultFramework to happen before this, so we put this BeforeEach // after NewDefaultFramework. </ins> BeforeEach(func() { c = f.ClientSet ns = f.Namespace.Name aggrclient = f.AggregatorClient }) <del> AfterEach(func() { cleanTest(c, aggrclient, ns) }) </del> It("Should be able to support the 1.7 Sample API Server using the current Aggregator", func() { // Make sure the relevant provider supports Agggregator framework.SkipUnlessServerVersionGTE(serverAggregatorVersion, f.ClientSet.Discovery())

helps but is not enough for the fix. After , I saw another failure: https://k8s- Before, all test after Aggregator test would fail. Now all tests before/after Aggregator tests are passing. However, during the Aggregator test waiting for namespace cleanup (10 min until timeout), all the other tests are failing. The reason is that, s run in LIFO order, s run in FIFO order. inserts an function to do namespace cleanup. In the current code, the namespace cleanup is inserted by before , it means that the test will try to delete namespace first, and then cleanup the api service. However, as known to us, if api service is not cleaned up, namespace deletion will wait forever.


Question: clientset "k8s.io/client-go/kubernetes" apiregistrationv1beta1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1beta1" aggregatorclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset" <ins> rbacv1beta1helpers "k8s.io/kubernetes/pkg/apis/rbac/v1beta1" </ins> "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" samplev1alpha1 "k8s.io/sample-apiserver/pkg/apis/wardle/v1alpha1"

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: cpuStats.UsageCoreNanoSeconds = &cstat.Cpu.Usage.Total } } <del> if info.Spec.HasMemory { </del> <ins> if info.Spec.HasMemory && cstat.Memory != nil { </ins> pageFaults := cstat.Memory.ContainerData.Pgfault majorPageFaults := cstat.Memory.ContainerData.Pgmajfault memoryStats = &statsapi.MemoryStats{

What happened: Kubelet had crashed one one of the heavily loaded instances in AWS EC2 cluster (r4.8xlarge). Logs show the following traceback: What you expected to happen: Kubelet not to crash. How to reproduce it (as minimally and precisely as possible): Reproduction unknown at this point. Anything else we need to know?: Environment: Kubernetes version (use ): Kubelet v.1.12.7; Control plane v.1.12.8 Cloud provider or hardware configuration: AWS/EC2 OS (e.g: ): CentOS Linux release 7.6.1810 (Core) Kernel (e.g. ): 5.0.4- (non-standard kernel) Install tools: Custom-made
/sig node
/assign
This seems like we don't check if Memory is not on latest stats
Nice debugging :)


Question: Process: s.HandleDeltas, } <del> s.controller = New(cfg) </del> func() { s.startedLock.Lock() defer s.startedLock.Unlock() <ins> s.controller = New(cfg) </ins> s.started = true }()

Observed on but I don't think it's the pr. Similar to but I don't think I can dupe the stack traces.


Question: "//test/e2e/framework/providers/azure:go_default_library", "//test/e2e/framework/providers/gce:go_default_library", "//test/e2e/framework/providers/kubemark:go_default_library", <ins> "//test/e2e/framework/providers/openstack:go_default_library", </ins> "//test/e2e/framework/testfiles:go_default_library", "//test/e2e/manifest:go_default_library", "//test/utils:go_default_library",

What happened: When specifying "openstack" as cloud-provider of e2e test, the test is failed like What you expected to happen: We can run e2e test with openstack cloud provider. How to reproduce it (as minimally and precisely as possible): Just kick the following command: ` Anything else we need to know?: This issue doesn't happen on v1.13, so this issue seems to happen during this dev cycle for v1.14. Environment: Kubernetes version (use ): master for v1.14 Cloud provider or hardware configuration: OpenStack OS (e.g: ): Ubuntu 16.04.5 LTS Kernel (e.g. ): 4.4.0-142-generic Install tools: kubeadm Others:
/sig openstack /sig testing
The exception raises at
The reason of this issue comes from
Ideally it is nice to put a new directory for registering the cloud provider.
/assign
I was wondering we could remove "openstack" provider specific tests and conditions from e2e tests, but there are storage tests which are related to OpenStack in e2e tests. So it is nice to add registration for OpenStack at this time.


Question: // so we don't start reloading too soon. err := utilwait.PollImmediate(iptablesFlushPollTime, iptablesFlushTimeout, func() (bool, error) { for i := 1; i < len(tables); i++ { <del> if exists, err := runner.chainExists(tables[i], canary); exists || isResourceError(err) { </del> <ins> if exists, err := runner.ChainExists(tables[i], canary); exists || isResourceError(err) { </ins> return false, nil } }

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: var unsupportedErr = errors.New("cAdvisor is unsupported in this build") <ins> func (cu *cadvisorUnsupported) Start() error { return unsupportedErr } </ins> func (cu *cadvisorUnsupported) DockerContainer(name string, req *cadvisorApi.ContainerInfoRequest) (cadvisorApi.ContainerInfo, error) { return cadvisorApi.ContainerInfo{}, unsupportedErr }

This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?

I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.


Question: if volPath != volumePath { t.Errorf("Got unexpected path: %s", volPath) } <del> if err := mounter.SetUp(volume.MounterArgs{}); err != nil { t.Errorf("Expected success, got: %v", err) </del> <ins> if config.volumeDirExists { if err := os.MkdirAll(volPath, perm); err != nil { t.Errorf("fail to create path: %s", volPath) } </ins> } // Stat the directory and check the permission bits <del> fileinfo, err := os.Stat(volPath) if !config.idempotent { if err != nil { if os.IsNotExist(err) { t.Errorf("SetUp() failed, volume path not created: %s", volPath) } else { t.Errorf("SetUp() failed: %v", err) } } if e, a := perm, fileinfo.Mode().Perm(); e != a { t.Errorf("Unexpected file mode for %v: expected: %v, got: %v", volPath, e, a) } } else if err == nil { // If this test is for idempotency and we were able // to stat the volume path, it's an error. t.Errorf("Volume directory was created unexpectedly") } </del> <ins> testSetUp(mounter, metadataDir, volPath) </ins> log := physicalMounter.GetLog() // Check the number of mounts performed during setup

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: bindClusterRBACRoleToServiceAccount(ctx, f, s, "cluster-admin") installSteps := []string{ <del> "echo "@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing/" >> /etc/apk/repositories", "&& apk add kubectl@testing gettext openssl", </del> <ins> "echo "@community http://dl-cdn.alpinelinux.org/alpine/edge/community/" >> /etc/apk/repositories", "&& apk add kubectl@community gettext openssl", </ins> "&& apk add --update coreutils", fmt.Sprintf("&& curl %s > gmsa.sh", gmsaWebhookDeployScriptURL), "&& chmod +x gmsa.sh",

[sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support can read and write file to remote SMB folder [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end May 24 2022 I think it is related to these setup steps No response_ /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: echo "!!! GNU sed is required.  If on OS X, use 'brew install gnu-sed'." fi <del> echo "+++ Versioning documentation and examples" # Update the docs to match this version. HTML_PREVIEW_PREFIX="https://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes" md_dirs=(docs examples) md_files=() for dir in "${md_dirs[@]}"; do mdfiles+=($( find "${dir}" -name "*.md" -type f )) done for doc in "${mdfiles[@]}"; do $SED -ri  -e '/<!-- BEGIN STRIP_FOR_RELEASE -->/,/<!-- END STRIP_FOR_RELEASE -->/d'  -e "s|(releases.k8s.io)/[^/]+|1/${NEW_VERSION}|"  "${doc}" # Replace /HEAD in html preview links with /NEW_VERSION. $SED -ri -e "s|(${HTML_PREVIEW_PREFIX}/HEAD)|${HTML_PREVIEW_PREFIX}/${NEW_VERSION}|" "${doc}" is_versioned_tag='<!-- TAG IS_VERSIONED -->' if ! grep -q "${is_versioned_tag}" "${doc}"; then echo "${is_versioned_tag}" >> "${doc}" fi done # Update API descriptions to match this version. $SED -ri -e "s|(releases.k8s.io)/[^/]+|1/${NEW_VERSION}|" pkg/api/v[0-9]*/types.go ${KUBE_ROOT}/hack/run-gendocs.sh ${KUBE_ROOT}/hack/update-swagger-spec.sh </del> <ins> echo "+++ Running ./versionize-docs" ${KUBE_ROOT}/build/versionize-docs.sh ${NEW_VERSION} </ins> git commit -am "Versioning docs and examples for ${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}" dochash=$(git log -n1 --format=%H)

If you try mimicking the release flow for a release on right now, it's busted: If I generate a dummy commit out of that failure, then run to see what gets generated, it's attempting to actually add the unversioned warnings back in: I was trying to write up a PR to enact something for , but this blocks the release piece for that. Which is fine, we have some time.
cc
Running gendocs should add the unversioned warnings, unless the doc has <!-- TAG IS_VERSIONED --. Do you know if mark-new-version that tag before failing?
Ah, I think I see what might be happening. This is the post- version of Notice the newline fun.
Sorry which newline are you talking about? :)
My PR fixes that
Ah, yes, cool. It would help if you flagged functional changes along more loudly. :)
Yea, sorry about that. Have updated the PR description to highlight each fix.


Question: package e2e_node import ( <del> "github.com/golang/glog" </del> "os/exec" <ins> "time" "github.com/golang/glog" </ins> ) const ( <del> busyBoxImage = iota </del> <ins> // Number of attempts to pull an image. maxImagePullRetries = 5 // Sleep duration between image pull retry attempts. imagePullRetryDelay = time.Second busyBoxImage        = iota </ins> hostExecImage netExecImage

Another image pull related failure: Looking at the docker logs, it looks like docker is getting a 404 pulling some of the images
The error message looks normal. Kubelet always inspects to check whether the image is present before pulling image. If the image is not present, docker will record 404 error in log. However, the wired thing is that, if we've already prepull images, the image should be present. :)
The logs are from May 2nd, when we didn't have the image prepulling. Closing the issue.


Question: package registry import ( <ins> "context" </ins> "testing" "time"

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: # Tests which kills or restarts components and/or nodes. DISRUPTIVE_TESTS=( <del> "AutoscalingsSuite.*scalescluster" </del> "DaemonRestart" "Etcdsfailure" "NodessResize"

Per our email discussion while getting 1.1 out the door. It looks like this never happened, though, which means we're not running autoscaling tests against GCE on the 1.1 branch, (we were running it on GKE with the 1.1-features job, but I'm removing that job in ). I'd once again like to (strongly) suggest that we mirror test suites running against release branch to jobs running against master, so that we avoid this problem of accidentally not running tests when we think we are. I'd much rather have a perhaps-unnecessary slow/HPA/whatever suite than continually try to figure out where slow/HPA/whatever tests are running, depending on if I'm looking at release jobs versus jobs running against master. cc
+1 We should be running the autoscaling tests in slow or continuous.


Question: if c.Debug != nil { c.Debug.Logf("Calling custom conversion of '%v' to '%v'", st, dt) } <del> args := []reflect.Value{sv.Addr(), dv.Addr(), reflect.ValueOf(scope)} ret := fv.Call(args)[0].Interface() // This convolution is necessary because nil interfaces won't convert // to errors. if ret == nil { return nil } return ret.(error) </del> <ins> return c.callCustom(sv, dv, fv, scope) </ins> } if !scope.flags.IsSet(AllowDifferentFieldTypeNames) && c.NameFunc(dt) != c.NameFunc(st) {

In Go you cannot take the address of a because the map implementation hides that from you. Our conversion library assumes that any reflected map value can have Addr() taken, which causes a panic when you try to do the following for PodInfo conversion: The conversion code tries to call Addr() on the reflect.Value (the map's value, ContainerStatus value object) which fails because it cannot have an address. I worked around it by registering a conversion function on *PodInfo and then manually converting the map.
Oh you mean the reflection code can't addr that? ok sounds like a bug. Non-reflection code can certainly do it:
That playground is taking the address of a value before it's in the struct. You can't take the address of m ----- Original Message -----
Right, because go does magical c++ style references there. But I thought the map copying code handled that case specially already?
I think I see what you're talking about. I'll make a test.
I repro'd. Will fix.
It was actually the source value that caused the trouble. Thanks for the report!
Sorry, should have been clearer. ----- Original Message -----
Note that this map violates our API conventions. It should be a list of named subobjects.
Yeah this field has always been problematic. It's still a bug regardless-- could have triggered copying other maps.


Question: apiserver.InstallSupport(m.muxHelper, m.rootWebService) apiserver.AddApiWebService(m.handlerContainer, c.APIPrefix, apiVersions) <ins> apiserver.InstallServiceErrorHandler(m.handlerContainer) </ins> // Register root handler. // We do not register this using restful Webservice since we do not want to surface this in api docs.

If go-restful detects a particular method is not allowed, it will return this error: That is preventing useful methods that resthandler would generate from making it to the client. Expected response:
triggered with
/cc
/cc
Is the right fix to update the client to generate the message versus changing the server to allow the method and report the error? My bias is to improve the client and not change the server as it makes reading our swagger doc easier to not see a bunch of verb unsupported entries. Sent from my iPhone
My bias is also towards not changing the server to keep swagger spec correct. By changing the client, do you mean changing kubectl? can you please clarify how is the json useful to client? It does not contain any useful extra information. On Feb 11, 2015 7:28 PM, "Derek Carr" wrote:
Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to. Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect.
This was an unintended side effect, but I think the old behavior was an unexpected luxury? I agree from the end-user perspective, the old message was nice, and it's great that the client was politely told that a given verb was not supported on the resource, but I don't agree that a server should appear to accept a method to only report that its not allowed. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:04:41 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to.

Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect. Reply to this email directly or view it on GitHub:
If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error?
I think so, this is only going to get worse when we support specific types of verbs on specific resources. In practice, it seems fine for kubectl to give a better error specific to the method not allowed response. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:26:08 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error? Reply to this email directly or view it on GitHub:
We could consider submitting a PR to go-restful to allow us to provide an error-generation function or interface implementation.
Yeah, that seems reasonable.
Opened
Copying text from : Some errors return as json and some as plain text. This causes overhead on client side when parsing the errors since need to recheck content type for every error in order not to parse plain text as json (some libraries fail on that). The errors message format should be standardized. Examples: The text/plain errors are from go-restful. cc
+1, thanks
The status objects are always of the default API version, however, rather than the version requested, because the request object isn't available to the error handler.
Rather than: we need:
Can we define an error handler on a web service?
At the moment, it's at the container level:
David and I talked about keeping the API lookup mechanism so we could use that in the error handler (apirequestinfo). We still need it for some use cases (proxies) so it could be used here to identify the version and proposed resource if matched.


Question: <ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "io/ioutil" "net" "strings" "sync" ) type command struct { cmd       string component string } func coreDump(dir string) { c, err := loadClient() if err != nil { fmt.Printf("Error creating client: %v", err) return } provider := testContext.Provider if !providerIs("gce", "gke") { fmt.Printf("Skipping SSH core dump, which is not implemented for %s", provider) return } // Get all nodes' external IPs. hosts, err := NodeSSHHosts(c) if err != nil { fmt.Printf("Error getting node hostnames: %v", err) return } cmds := []command{ {"cat /var/log/kubelet.log", "kubelet"}, {"cat /var/log/kube-proxy.log", "kube-proxy"}, } logCore(cmds, hosts, dir, provider) // I wish there was a better way to get the master IP... config, err := loadConfig() if err != nil { fmt.Printf("Error loading config: %v") } ix := strings.LastIndex(config.Host, "/") master := net.JoinHostPort(config.Host[ix+1:], "22") cmds = []command{ {"cat /var/log/kube-apiserver.log", "kube-apiserver"}, {"cat /var/log/kube-scheduler.log", "kube-scheduler"}, {"cat /var/log/kube-controller-manager.log", "kube-controller-manager"}, } logCore(cmds, []string{master}, dir, provider) } func logCore(cmds []command, hosts []string, dir, provider string) { wg := &sync.WaitGroup{} // Run commands on all nodes via SSH. for _, cmd := range cmds { fmt.Printf("SSH'ing to all nodes and running %sn", cmd.cmd) for _, host := range hosts { go func() { defer wg.Done() wg.Add(1) logfile := fmt.Sprintf("%s/%s-%s.log", dir, host, cmd.component) fmt.Printf("Writing to %s.n", logfile) stdout, stderr, _, err := SSH(cmd.cmd, host, provider) if err != nil { fmt.Printf("Error running command: %vn", err) } if err := ioutil.WriteFile(logfile, []byte(stdout+stderr), 0777); err != nil { fmt.Printf("Error writing logfile: %vn", err) } }() } } wg.Wait() } </ins>

We need to be able to diagnose e2e test failures after the relevant test cluster has been shut down. Right now this almost works, but not quite. Fluentd can currently only be configured to send log files to one place, for example: Cloud Logging, or (running as a pod in the e2e test cluster, and saved to local disk on the node). With the current configuration in , logs typically go to ElasticSearch (and there is an e2e test to check that that works correctly), and tears down the e2e test cluster after each test run, so the logs are deleted. The proposed solution is to have fluentd configurable to send to multiple log collector backends, and configure it as such to both Google Cloud Logging and ElasticSearch for CI e2e test runs to facilitate easier debugging. Satnam6502@ has offered to sort this out.
Originally it was possible to send logs to both Elasticsearch and Cloud Logging but at some point this got made into a binary choice of one or the other (by or ?). I propose to undo this unless there are strong objections. : what do you think?
I didn't turn it into a binary choice. :) , Satnam Singh wrote:
Yeah, pretty sure it was Joe. I am sure it was a wide decision at the time.
Would be nice to have these at maximum verbosity because when we need them we need them all, IIRC we currently only get V(2) on a kube-up cluster
I see this is marked as a post v1.0 milestone? If we don't want this urgently then I will un-assign myself for now.
There is a valid argument that we would not block the 1.0 release for lack of these persisted test logs, and hence the v1.0 milestone was tentatively removed. There is a counter argument that we stand little chance of being able to debug the existing and still-to-be-found flaky test failures in the absence of log files, and that one or more of those bugs, if not fixed, will block the launch anyway. I'd say that if there is nothing on the v1.0 milestone that you can sensibly work on, this one is fair game.

I would find it useful, and I suppose this should be on par with writing tests (of which we have a few 1.0: )
I'm not sure we'll be able to convince folks to move this to 1.0, but feel free to work on this, as it seems very valuable.
I thought about changing our setup to allow both Elasticsearch and Cloud Logging to be targeted. I've been a little hesitant because this could lead to a breakage (esp. for GKE) and we are getting close to 1.0 ...
/cc who might have memories about a GKE breakage caused by changes in env vars used for logging.
I'd love to see this fixed soon to help debug flakes, but yes, it'd be nice to at least be cc'ed on env var changes.
Try again to have it to the high priority list, as this is making debugging the system very difficult. cc Who expressed interest.
How do we access those logs? I tried to find them for gce-scalability jenkins suite, but didn't find it.


Question: "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/util/slice" "k8s.io/kubernetes/pkg/volume" <del> utilptr "k8s.io/utils/pointer" </del> <ins> "k8s.io/utils/ptr" </ins> ) func TestLoadPodFromFile(t *testing.T) {

projected volumes exclude ephemeral containers in calculating fs user because "SecurityContext is not allowed" : This is no longer technically correct -- ephemeral containers are allowed to specify security context -- but it's a no-op because is guaranteed to be empty at pod creation when fs user is calculated. Still, the comment is wrong so let's clean it up to avoid confusion. /kind cleanup
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /assign
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: import ( "reflect" <ins> "strconv" </ins> "time" "encoding/json"

This is mostly motivated by petset. Currently we create a governing service that is responsible for the network identity of each pet. The requirements of this governing service differ from a normal service in the following ways: have internal protocols to notice peer health, and a list of different checks (eg: ). In kube, peers use DNS to communicate. So we don't want a readiness check failure to remove the pet from DNS and abruptly cut it off from the set. "governing service" is headless so kube-proxy doesn't need to NAT the service ip. will not be using this Service, so it doesn't matter that unready pets show up in DNS under the svcName sub domain. In addition to these, there's one important reason some pets need to have DNS before readiness. To provide the guarantees required to safely bootstrap a pet, we need: 0 starts before pet1.. 't start pet1 till pet0 is running/ready pet1 starts, it sees pet0 in DNS The easiest way to guarantee 3 today is to do one of: poll in the entrypoint of pet0, till it sees itself poll in the init container of pet0, till it sees itself poll in the init containers of pet {1..N-1} till it sees all previous pets Third option is not a great idea because we don't know who else is supposed to exist, and for the first 2 to work with a readiness probe, we need a DNS record before the Service is ready.
I have a wip that does this through an annotation:
+1. this is useful even in traditional n rc's and n+1 services setup for clustered systems like n nodes zookeeper/etcd. Currently, there isn't a easy way to specify a readiness check in such setups, because the check in these cases would look for whether the pod member has joined the cluster successfully as follower, which wouldn't work as they discover each other through them appearing in service endpoints. this prevents one from being able to do a rolling upgrade of cluster members safely, because there is no readiness feedback after a member pod is upgraded.
Suggestion to hold this for alpha in 1.5, and in 1.6 spend a bit of time talking about how services are best intermixed with stateful sets (there's a lot of correlation between them).

That avoids us breaking people in 1.5 and gets a bit more think time.
SGTM
See also
Can we claim that if a pod is requesting pet DNS (i.e DNS name via the hostname/subdomain annotation) it never wants kube-style readiness, because it has its own internal protocol for peer health? If that's true we can get rid of the annotation and teach kubedns to always serve records for pet DNS ignoring readiness. My intuition tells me the answer to that question is no (i.e there exist pets that use DNS for discovery, and don't have an internal protocol for readiness, so they must use kube readiness to remove themselves from the peer group), and so the ability to toggle it on/off is useful, but I also want others to consider it.
I'd love to nix an annotation but I have not the expertise to answer this one.
I don't have a strong feel either way - I'd say readiness controls on a service feel orthogonal to independent names though. Since we don't have per port readiness, we need to consider that at some point we might need more sophistication on services or pods to control membership (unless we make label flipping easier). I would like to see us tease apart the "leader / follower" service member concept a bit more because that might inform how we handle readiness (seems like a separate bit, but we may want atomic changes).
They are somewhat orthogonal, but you presumably only want to discover healthy peers, unless you have a way to probe every peer you discover for health. Applying this policy based on endpoint info also feels a little weird, because it's harder to create overlapping Servics with different behavior. I think doing the common thing always and by default is fine, as long as we can annotate our way back to what we have today, but what form would the opposite of today's Service level annotation take? i.e something that says "even though my endpoints need pet names, apply the regular rediness behavior to their DNS records"
How bad would it be to start without an annotation, assume that pet-names == tolerate-unready, and see if anyone screams? Who should we consult?
SGTM, that's also what I was prposing in the previous comment
I guess the question is why do we even want to do that? Why make a change that couples logic vs leaving as is with decoupled logic?

We have people who want unready endpoints for deployments to form gossip pools - so I guess that counts as a use case where pet names is not coupled to unready. Is leaving the annotation on for another release really a problem? , Prashanth B <:
That's exactly the kind of use case I was looking for, something that doesn't require pet names. I get the feeling there will be other apps but I couldn't produce an example.
JBoss EAP uses it to form initial quorum for jgroups, which isn't a strong consensus system. So they use deployments and a minimum set to form an initial quorum and deal with split brain like jgroups normally does (infinispan has multiple consistent modes, some of which might prefer loose pools). Arguably smaller sets and some infinispan quorums might prefer statefulsets, but those use cases are also more tolerant of partitions in general. , Prashanth B <:
we are using tolerate-unready-endpoints annotation in our zookeeper deployments (not using pet/stateful set). Without unready endpoints, it is impossible to define a meaningful pod readiness check for zk, and without a readiness check, it is hard to do rolling updates (one ensemble member at a time).
The question is whether there's a need to have StatefulSets without tolerate-unready-endpoints. It sounds like maybe yes, which makes me grumpy, but I'll acquiesce if Clayton and Prashanth say so. , ravilr <:
isn't it enough to establish a need to tolerate-unready without statefulsets?
I have two services pointing to my etcd stateful set - one that is endpoints (peers) and tolerates unready. I have one that is for naive clients that uses readiness. On Dec 1, 2016, at 2:59 AM, Prashanth B <: isn't it enough to establish a need to tolerate-unready without statefulsets? — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
fine.
/sig network
Right. This is exactly why etcd operator uses the alpha feature. Can we make it beta in 1.8 release?
/cc
We had discussed promoting since we need it for StatefulSet and no one has come up with anything better.

We should make a decision and include it in our GA roadmap for workload controllers.
Be sure to include sig-network On Jul 6, 2017, at 2:40 PM, Anthony Yeh <: had discussed promoting tolerate-unready-endpoints since we need it for StatefulSet and no one has come up with anything better. We should make a decision and include it in our GA roadmap for workload controllers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
This is also useful for ingresses that need to retain the source IP using OnlyLocal node ports, which is important for firewalling with things like Calico. The problem that we see now is when a deployment occurs, the pod is removed from the node port before the PreStop completes, which doesn't give the upstream AWS load balancer time for the pod to fail out using normal health checks.
/close
Headless Services are a thing now