Question: // FlattenListVisitor flattens any objects that runtime.ExtractList recognizes as a list // - has an "Items" public field that is a slice of runtime.Objects or objects satisfying <del> // that interface - into multiple Infos. An error on any sub item (for instance, if a List // contains an object that does not have a registered client or resource) will terminate // the visit. // TODO: allow errors to be aggregated? </del> <ins> // that interface - into multiple Infos. Returns nil in the case of no errors. // When an error is hit on sub items (for instance, if a List contains an object that does // not have a registered client or resource), returns an aggregate error. </ins> type FlattenListVisitor struct { visitor Visitor typer   runtime.ObjectTyper

What happened: When using a yaml file for input, silently fails at the first missing resource, not returning any further results. What you expected to happen: Searching with cli arguments properly ignores missing resources and continues the query, eg: Would expect similar results with a file input, eg: How to reproduce it (as minimally and precisely as possible): With an input file of : the command fails at apparently the first missing resource: Note: that rearranging the inputs in (so that the missing resource is last) produces the "expected" output: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: local OS (e.g: ): fedora 30 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: x-ref:
/sig cli
/assign
Got around to reproducing it and can confirm that I'm getting the same behavior. I'm looking into fixing it now
thanks, any updates on this?
I've learned that in the case of a fake entry, kubectl stops sending API get requests altogether. Learned by running kubectl get -f --ignore-not-found --v=10, the result is shown below. Till now I've been looking into the cobra command framework but now I'm trying to find out where kubectl is making these API calls from(working on finding out where this connection is happening from Slack+debugging, I've asked a more detailed question here ). <img width="1440" alt="Screen Shot 2019-08-12 at 4 13 41 PM" src="https://user-"
that's an interesting find, definitely helpful. When I looked into this I got to the section of get's main run function, and it seems like it should be continuing on any errors (see ) but maybe the ResourceBuilder is actually stopping when it hits a 404?
This behavior comes from where the not found errors are being ignored and where the input list is flattened. Errors are ignored here: While flattening occurs here: As the visitor iterates through the objects in the list if an error occurs, such as not found, it will stop iterating over the objects in the list and return that not found error. The logic will then ignore it and continue.

Any objects that came before the not found object in the list will have been visited and will then be included in the output. To make the behavior consistent between the two ways of getting the set of objects, the would need to be updated to support filtering errors.
Just submitted a PR adding that functionality to the FlattenListVisitor. The behavior is now consistent between the two methods.


Question: // GetCandidateZonesForDynamicVolume retrieves  a list of all the zones in which nodes are running // It currently involves querying all instances func (c *Cloud) GetCandidateZonesForDynamicVolume() (sets.String, error) { <del> // We don't currently cache this; it is currently used only in volume // creation which is expected to be a comparatively rare occurrence. // TODO: Caching / expose v1.Nodes to the cloud provider? // TODO: We could also query for subnets, I think // Note: It is more efficient to call the EC2 API twice with different tag // filters than to call it once with a tag filter that results in a logical // OR. For really large clusters the logical OR will result in EC2 API rate // limiting. instances := []*ec2.Instance{} baseFilters := []*ec2.Filter{newEc2Filter("instance-state-name", "running")} </del> <ins> zones := sets.NewString() </ins> <del> filters := c.tagging.addFilters(baseFilters) di, err := c.describeInstances(filters) </del> <ins> // TODO: list from cache? nodes, err := c.kubeClient.CoreV1().Nodes().List(metav1.ListOptions{}) </ins> if err != nil { <ins> klog.Errorf("Failed to get nodes from api server: %#v", err) </ins> return nil, err } <del> instances = append(instances, di...) if c.tagging.usesLegacyTags { filters = c.tagging.addLegacyFilters(baseFilters) di, err = c.describeInstances(filters) if err != nil { return nil, err </del> <ins> for _, n := range nodes.Items { if !c.isNodeReady(&n) { klog.V(4).Infof("Ignoring not ready node %q in zone discovery", n.Name) continue </ins> } <del> instances = append(instances, di...) } if len(instances) == 0 { return nil, fmt.Errorf("no instances returned") } zones := sets.NewString() for _, instance := range instances { // We skip over master nodes, if the installation tool labels them with one of the well-known master labels // This avoids creating a volume in a zone where only the master is running - e.g. #34583 // This is a short-term workaround until the scheduler takes care of zone selection master := false for _, tag := range instance.Tags { tagKey := aws.StringValue(tag.Key) if awsTagNameMasterRoles.Has(tagKey) { master = true </del> <ins> // In some cluster provisioning software, a node can be both a minion and a master. Therefore we white-list // here, and only filter out node that is not minion AND is labeled as master explicitly if c.isMinionNode(&n) || !c.isMasterNode(&n) { if zone, ok := n.Labels[v1.LabelZoneFailureDomain]; ok { zones.Insert(zone) } else { klog.Warningf("Node %s does not have zone label, ignore for zone discovery.", n.Name) </ins> } <ins> } else { klog.V(4).Infof("Ignoring master node %q in zone discovery", n.Name) </ins> } <ins> } </ins> <del> if master { klog.V(4).Infof("Ignoring master instance %q in zone discovery", aws.StringValue(instance.InstanceId)) continue } </del> <ins> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil } </ins> <del> if instance.Placement != nil { zone := aws.StringValue(instance.Placement.AvailabilityZone) zones.Insert(zone) </del> <ins> // isNodeReady checks node condition and return true if NodeReady is marked as true func (c *Cloud) isNodeReady(node *v1.Node) bool { for _, c := range node.Status.Conditions { if c.Type == v1.NodeReady { return c.Status == v1.ConditionTrue </ins> } } <ins> return false } // isMasterNode checks if the node is labeled as master func (c *Cloud) isMasterNode(node *v1.Node) bool { // Master node has one or more of the following labels: // // 	kubernetes.io/role: master //	node-role.kubernetes.io/master: "" //	node-role.kubernetes.io/master: "true" if val, ok := node.Labels[labelKeyNodeMaster]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMasterRole { return true } return false } </ins> <del> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil </del> <ins> // isMinionNode checks if the node is labeled as minion func (c *Cloud) isMinionNode(node *v1.Node) bool { // Minion node has one or more oof the following labels: // // 	kubernetes.io/role: "node" //	node-role.kubernetes.io/compute: "true" //	node-role.kubernetes.io/node: "" if val, ok := node.Labels[labelKeyNodeMinion]; ok && val != "false" { return true } else if val, ok := node.Labels[labelKeyNodeCompute]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMinionRole { return true } return false </ins> } // GetZone implements Zones.GetZone

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: pod := entrypointTestPod() pod.Spec.Containers[0].Command = []string{"/ep-2"} <del> testContainerOutput("override command", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override command", c, pod, []string{ </ins> "[/ep-2]", }) })

Testing how kubelet client rotation works, I discovered that on startup kubelet uses the cert in () and not the new cert in () This prevent kubelet to start kubelet uses the latest client certificate a 1 node cluster with kubespray (kubeadm should do) ntpd/chronyd the date 11 months in the future master certs kubelet that you have a new client cert the date 11 months more in the future and restart kubelet kubelet fails to start because it uses the cert embedded in instead of Both and are set, if I remove kubelet boots but fails to renew the cert Here an easy fix: Also I noticed that is RSA where is EC/P-256 Kubernetes version (use ): v1.18.15 (I can retest in more recent version but haven't seen anything in the release notes) Cloud provider or hardware configuration: VM OS (e.g: ): Kernel (e.g. ): Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle (not sure if this is more kubelet or kubeadm)
is the embedding certificates? see the second warning under here:
you can call on the same node to make the point to the rotatable file. newer versions of kubeadm do this change automatically.
i don't see a core bug here. i missed that you are using kubespray, which uses kubeadm. please report any issues with the kubespray issue tracker first: /kind support /close
Closing this issue. <detailsIn response to : i don't see a core bug here. please triage any issues with the kubespray issue tracker first: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks kubespray indeed doesn't call anywhere, I'll go look at how to fix that
like i said, newer versions of do this automatically.

this might be the result of an old cluster being upgrade using but the is remaining stale. the manual step in or calling the on a node is required.
It was a fresh deployment of k8s 1.18 / kubespray 2.15, I'm doing a fresh deployment of k8s 1.20 / kubespray master right now, but maybe kubespray is using kubeadm in weird ways / include old workarounds that leave us with having embedded certs instead of pointing to .
Found the culprit: Kubespray regenerate the conf files using kubeadm to make sure the server is up to date Thanks for the help and sorry for the spam
it would make sense to just include the kubelet-finalize phase in there too.
I prefer to just change the server using one of lineinfile/sed/kubectl so next time there is a new step like kubelet-finalize we don't break again.


Question: name: "net.ipv4.tcp_keepalive_time", // refer to https://github.com/torvalds/linux/commit/13b287e8d1cad951634389f85b8c9b816bd3bb1e. kernel: "4.5", <ins> }, { // refer to https://github.com/torvalds/linux/commit/1e579caa18b96f9eb18f4f5416658cd15f37c062. name:   "net.ipv4.tcp_fin_timeout", kernel: "4.6", }, { // refer to https://github.com/torvalds/linux/commit/b840d15d39128d08ed4486085e5507d2617b9ae1. name:   "net.ipv4.tcp_keepalive_intvl", kernel: "4.5", }, { // refer to https://github.com/torvalds/linux/commit/9bd6861bd4326e3afd3f14a9ec8a723771fb20bb. name:   "net.ipv4.tcp_keepalive_probes", kernel: "4.5", </ins> }, }

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: "Currently only applies to long-running requests.") fs.IntVar(&s.MasterCount, "apiserver-count", s.MasterCount, <del> "The number of apiservers running in the cluster.") </del> <ins> "The number of apiservers running in the cluster, must be a positive number.") </ins> // See #14282 for details on how to test/try this option out. // TODO: remove this comment once this option is tested in CI.

If kube-apiserver is run with , odd things happen e.g. We probably should validate that 0.
I'd like to work on this:)


Question: return fmt.Errorf("error storing status for daemon set %#v: %v", ds, err) } <ins> // Resync the DaemonSet after MinReadySeconds as a last line of defense to guard against clock-skew. if ds.Spec.MinReadySeconds > 0 && numberReady != numberAvailable { dsc.enqueueDaemonSetAfter(ds, time.Duration(ds.Spec.MinReadySeconds)*time.Second) } </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: RollingUpdate daemonset got stucked, leaves a few pod not updated. Daemonset's status are as follows: What you expected to happen: All pods managed by daemonset upgrade succeed. How to reproduce it (as minimally and precisely as possible): minReadySeconds for deamonset -controller's clock time is a little slower than node. Anything else we need to know?: related issue and pr Environment: the latest k8s version also has this bug
/sig apps
/assign
Please rename our production daemonset name to "foo-daemonset"


Question: func makeLinkVariables(service api.Service, machine string) []api.EnvVar { prefix := makeEnvVariableName(service.ID) <del> var port string if service.ContainerPort.Kind == util.IntstrString { port = service.ContainerPort.StrVal } else { port = strconv.Itoa(service.ContainerPort.IntVal) </del> <ins> protocol := "TCP" if service.Protocol != "" { protocol = service.Protocol </ins> } <del> portPrefix := prefix + "_PORT_" + makeEnvVariableName(port) + "_TCP" </del> <ins> portPrefix := fmt.Sprintf("%s_PORT_%d_%s", prefix, service.Port, strings.ToUpper(protocol)) </ins> return []api.EnvVar{ { Name:  prefix + "_PORT", <del> Value: fmt.Sprintf("tcp://%s:%d", machine, service.Port), </del> <ins> Value: fmt.Sprintf("%s://%s:%d", strings.ToLower(protocol), machine, service.Port), </ins> }, { Name:  portPrefix, <del> Value: fmt.Sprintf("tcp://%s:%d", machine, service.Port), </del> <ins> Value: fmt.Sprintf("%s://%s:%d", strings.ToLower(protocol), machine, service.Port), </ins> }, { Name:  portPrefix + "_PROTO", <del> Value: "tcp", </del> <ins> Value: strings.ToLower(protocol), </ins> }, { Name:  portPrefix + "_PORT",

We have seen that on VirtualBox in particular kubelet picks first interfaces, which is usually NATed and is of no use. In particular the issue comes up with service IP, which gets mapped to endpoint on the NATed interface.
If this is still a problem, can you explain more - I don't quite understand - and suggest heuristics for better solutions?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return nil } <del> func cascadingDeleteResource(info *resource.Info, out io.Writer, shortOutput bool) error { </del> <ins> func cascadingDeleteResource(info *resource.Info, out io.Writer, shortOutput bool, gracePeriod int) error { </ins> falseVar := false deleteOptions := &metav1.DeleteOptions{OrphanDependents: &falseVar} <del> return deleteResource(info, out, shortOutput, deleteOptions) </del> <ins> return deleteResource(info, out, shortOutput, deleteOptions, gracePeriod) </ins> } <del> func deleteResource(info *resource.Info, out io.Writer, shortOutput bool, deleteOptions *metav1.DeleteOptions) error { </del> <ins> func deleteResource(info *resource.Info, out io.Writer, shortOutput bool, deleteOptions *metav1.DeleteOptions, gracePeriod int) error { </ins> if err := resource.NewHelper(info.Client, info.Mapping).DeleteWithOptions(info.Namespace, info.Name, deleteOptions); err != nil { return cmdutil.AddSourceToErr("deleting", info.Source, err) } <del> printDeletion(info, out, shortOutput) </del> <ins> printDeletion(info, out, shortOutput, gracePeriod) </ins> return nil } // deletion printing is special because they don't have an object to print.  This logic mirrors PrintSuccess <del> func printDeletion(info *resource.Info, out io.Writer, shortOutput bool) { </del> <ins> func printDeletion(info *resource.Info, out io.Writer, shortOutput bool, gracePeriod int) { </ins> operation := "deleted" groupKind := info.Mapping.GroupVersionKind kindString := fmt.Sprintf("%s.%s", strings.ToLower(groupKind.Kind), groupKind.Group)

IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


Question: done set -- "${testcases[@]+${testcases[@]}}" <ins> # TODO: this should probably be refactored to avoid code duplication with the # coverage version. </ins> if [[ $iterations -gt 1 ]]; then if [[ $# -eq 0 ]]; then set -- $(kube::test::find_dirs)

Right now, we execute unit tests in Travis with , and you can see the results on each line: but we don't have an easy place to go browse our code and figure out what's covered and what isn't. Is there some awesome way to do this? At the very least, can we get the actual line-by-line coverage results integrated into our CI build process so we can poke at these somehow?
cc
I'm willing to take a look into this to at least see if something is possible.
Looking into using Coveralls; there are a few other teams at Google already using this. Integration with Travis and Go are supported with the goveralls package:
Unit test coverage can now be viewed at - both master push builds and PRs will be shown there; additionally, Coveralls will update the status and post comments on PRs with coverage stats. (If we find these too spammy, we can disable either or both.) The top-level also has a badge/link to the coverage dashbaord.


Question: - `--port` - `--address` The insecure port flags `--port` may only be set to 0 now. <ins> - Kube-scheduler: the `--port` and `--address` flags have no effect and would be removed in v1.24. The insecure port flags `--port` may only be set to 0 now. </ins> Also `metricsBindAddress` and `healthzBindAddress` fields from `kubescheduler.config.k8s.io/v1beta1` are no-op and expected to be empty. Removed in `kubescheduler.config.k8s.io/v1beta2` completely. ([#96345](https://github.com/kubernetes/kubernetes/pull/96345), [@ingvagabund](https://github.com/ingvagabund)) <ins> In addition, please be careful that: - kube-scheduler MUST start with `--authorization-kubeconfig` and `--authentication-kubeconfig` correctly set to get authentication/authorization working. - liveness/readiness probes to kube-scheduler MUST use HTTPS now, and the default port has been changed to 10259. - Applications that fetch metrics from kube-scheduler should use a dedicated service account which is allowed to access nonResourceURLs `/metrics`. ([#96345](https://github.com/kubernetes/kubernetes/pull/96345), [@ingvagabund](https://github.com/ingvagabund)) [SIG Cloud Provider, Scheduling and Testing] </ins> - Feature-gate `VolumeSubpath` has been deprecated and cannot be disabled. It will be completely removed in 1.25 ([#105474](https://github.com/kubernetes/kubernetes/pull/105474), [@mauriciopoppe](https://github.com/mauriciopoppe)) - Kubeadm: add a new `output/v1alpha2` API that is identical to the `output/v1alpha1`, but attempts to resolve some internal dependencies with the `kubeadm/v1beta2` API. The `output/v1alpha1` API is now deprecated and will be removed in a future release. ([#105295](https://github.com/kubernetes/kubernetes/pull/105295), [@neolit123](https://github.com/neolit123)) - Kubeadm: add the kubeadm specific, Alpha (disabled by default) feature gate UnversionedKubeletConfigMap. When this feature is enabled kubeadm will start using a new naming format for the ConfigMap where it stores the KubeletConfiguration structure. The old format included the Kubernetes version - "kube-system/kubelet-config-1.22", while the new format does not - "kube-system/kubelet-config". A similar formatting change is done for the related RBAC rules. The old format is now DEPRECATED and will be removed after the feature graduates to GA. When writing the ConfigMap kubeadm (init, upgrade apply) will respect the value of UnversionedKubeletConfigMap, while when reading it (join, reset, upgrade), it would attempt to use new format first and fallback to the legacy format if needed. ([#105741](https://github.com/kubernetes/kubernetes/pull/105741), [@neolit123](https://github.com/neolit123)) [SIG Cluster Lifecycle and Testing]

It looks like default HTTPS port for kube-scheduler has changed from to between 1.22.4 and 1.23.0 releases and it has not been mentioned in the changelog. EDIT: Actually, I see now that 1.22.4 listens on , but also on and 1.23.0 just stopped listening on . Breaking changes to be included in the changelog. With kube-prometheus-stack installed, with kube-scheduler scraping configured, upgrade kube-scheduler from 1.22.4 to 1.23.0. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release scheduling
was the change, and included a detailed release note. Did that not get pulled into the release notes automatically? /sig docs release
can you help update the 1.23 release notes with the content from in the "Action required" section?
/assign ingvagabund
I see now that it has indeed been mentioned for , here: but not for .
weird, v1.23.0 is supposed to roll up all the intermediate notes.
Is it sufficient to manually edit and extend
manual edit is fine, hoist it to the section


Question: "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait"

Which jobs are failing: ci-kubernetes-e2e-gce-new-master-upgrade-master-parallel Which test(s) are failing: [sig-apps] Deployment deployment should support rollback Since when has it been failing: 06-22 - 17:13 PDT Testgrid link: Reason for failure: Anything else we need to know: /priority critical-urgent /kind failing-test /sig apps
I will look in test together with
/milestone v1.16
stopped serving those APIs in 1.16. The test in the release-1.15 branch should skip if the server it is testing against does not serve the deprecated extensions/v1beta1 API
closing
and did not completely resolve the failing tests here or in /reopen
Reopened this issue. <detailsIn response to : and did not completely resolve the failing tests here or in /reopen </details
this job is running 1.14-level tests against master (1.16). the test should be updated to run 1.15 against master
Possible issue here is that we may have misconfigured upgrade/downgrade jobs (upgrading/downgrading to incorrect k8s versions). For full discussion see
/assign
Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close
Closing this issue. <detailsIn response to : Ultimately, the issue with this and was the the Kubernetes release markers did not point to the releases we expected. This was fixed during the recent 1.16 alpha release. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: normalizeTimeStamp(&condition.LastTransitionTime) } <del> // update container statuses for i := range status.ContainerStatuses { cstatus := &status.ContainerStatuses[i] normalizeContainerState(&cstatus.State) normalizeContainerState(&cstatus.LastTerminationState) </del> <ins> normalizeContainerStatuses := func(containerStatuses []v1.ContainerStatus) { for i := range containerStatuses { cstatus := &containerStatuses[i] normalizeContainerState(&cstatus.State) normalizeContainerState(&cstatus.LastTerminationState) } </ins> } <del> // Sort the container statuses, so that the order won't affect the result of comparison </del> <ins> normalizeContainerStatuses(status.ContainerStatuses) </ins> sort.Sort(kubetypes.SortedContainerStatuses(status.ContainerStatuses)) <del> // update init container statuses for i := range status.InitContainerStatuses { cstatus := &status.InitContainerStatuses[i] normalizeContainerState(&cstatus.State) normalizeContainerState(&cstatus.LastTerminationState) } // Sort the container statuses, so that the order won't affect the result of comparison </del> <ins> normalizeContainerStatuses(status.InitContainerStatuses) </ins> kubetypes.SortInitContainerStatuses(pod, status.InitContainerStatuses) <ins> normalizeContainerStatuses(status.EphemeralContainerStatuses) sort.Sort(kubetypes.SortedContainerStatuses(status.EphemeralContainerStatuses)) </ins> return status }

is not normalized in in the status manager: This does not seem to cause any user facing problem because is sorted before passed to the status manager and timestamps are normalized at marshaling to create a patch: However, an unexpected behavior is caused internally because timestamps are not normalized. If the log verbosity is 3 or larger, the following messages are logged periodically in kubelet when an ephemeral container exists: This message is emitted here: After returns , is called. This does not look so harmful because gets eventually: API is called once unnecessarily in , though: It might be better to normalize ephemeral container statuses, at least timestamps, in the status manager to avoid this unexpected behavior. Set the log verbosity of kubelet to 3 or larger. Then, create an ephemeral container like . No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
This is a bug. Can you submit a PR to fix it? If not, I'd be happy to work on it
If you are interested, please feel free to assign yourself.
thx
/assign
/triage accepted /priority important-longterm


Question: return false, nil } <del> completionMode := string(batch.NonIndexedCompletion) if isIndexedJob(&job) { completionMode = string(batch.IndexedCompletion) } </del> <ins> completionMode := getCompletionMode(&job) </ins> action := metrics.JobSyncActionReconciling defer func() {

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: select { case event := <-eventChan: return event == expected <del> case <-time.After(2 * time.Second): </del> <ins> case <-time.After(wait.ForeverTestTimeout): </ins> t.Fatalf("Timed out while waiting for registration status %v", expected) }

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: func getObjectKind(object interface{}) (schema.GroupVersionKind, []error) { var listErrors []error <del> fields := object.(map[string]interface{}) if fields == nil { </del> <ins> fields, ok := object.(map[string]interface{}) if !ok || fields == nil { </ins> listErrors = append(listErrors, errors.New("invalid object to validate")) return schema.GroupVersionKind{}, listErrors }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig cli What happened: I accidentally tried to apply an invalid v1.List with : kubectl responded by crashing with a panic: What you expected to happen: An error message to be printed, perhaps of the "invalid List item: []". How to reproduce it (as minimally and precisely as possible): See description. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS 7.4 (1708) Kernel (e.g. ): Linux staging-head-2 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
I would try to fix it.
Can you add --validate=true in command line, then have a try?
cc


Question: } // Start resourceMonitor only in small clusters. <del> if len(nodes.Items) <= maxNodesToCheck { </del> <ins> if numNodes <= maxNodesToCheck { </ins> resourceMonitor = e2ekubelet.NewResourceMonitor(f.ClientSet, e2ekubelet.TargetContainers(), containerStatsPollingInterval) resourceMonitor.Start() }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: [] [sig-node] kubelet [] [sig-node] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. Since when has it been failing: 2019-09-26 Testgrid link: Reason for failure: Anything else we need to know: /cc
is there someone who can pick this up since it's constantly failing in the master-informing? Thanks :)
Could you help us here?
Looked into the test logs, and it seems like the in was accidentally removed in , which caused its value to be 0. As the current log is showing: And last succeeded test log is: I'll send a PR to fix this. /cc
It seems that this PR didn't help. This run already contains this PR and it failed:
/milestone v1.17 /priority critical-urgent
I0926 12:39:42.660] I0926 12:35:26. ] Trace[]: "Reflector ListAndWatch" (started: 2019-09-26 12:33:26. +0000 UTC m=+1132.) (total time: 2m0.000301191s):
Seems like the test is timing out creating the pods and reflector to watch those pods.
Looks like the section of code taking a long time is
Checked KAS log and seeing about 7000 requests of the form I1010 12:26:59. 1 ] GET /api/v1/nodes/gce-scale-cluster-minion-group-1-4p2m:/proxy/stats/summary: (9.753712ms) 200 [ (linux/amd64) kubernetes/ -- [] [sig-node] kubelet [] [sig-node] Clean up pods on node kubelet should be able to delete 10 pods per node in 1m0s. 146.148.73.158:] between replicationcontroller being created and the pod list/watch request.
The is only supposed to run if there are less than 10 nodes in the cluster. However, the situation doesn't hold with this change: (or ) was changed to .

While the e2e test only uses 10 () nodes from the cluster, the resource monitor is querying all the nodes in the cluster. Thus the client qps limit was exceeded and caused the reflector to be throttled. I'll send a fix
/assign


Question: if disk.Lun != nil && (disk.Name != nil && diskName != "" && strings.EqualFold(*disk.Name, diskName)) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && strings.EqualFold(*disk.Vhd.URI, diskURI)) || (disk.ManagedDisk != nil && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { <del> // found the disk klog.V(2).Infof("azureDisk - find disk: lun %d name %q uri %q", *disk.Lun, diskName, diskURI) return *disk.Lun, nil </del> <ins> if disk.ToBeDetached != nil && *disk.ToBeDetached { klog.Warningf("azureDisk - find disk(ToBeDetached): lun %d name %q uri %q", *disk.Lun, diskName, diskURI) } else { // found the disk klog.V(2).Infof("azureDisk - find disk: lun %d name %q uri %q", *disk.Lun, diskName, diskURI) return *disk.Lun, nil } </ins> } } return -1, fmt.Errorf("cannot find Lun for disk %s", diskName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We found that cloud provider tries to attach a disk, it totally costs 121s in the end one minute(disk still in being detaching state), azure cloud provider tries to delete that disk and it succeeded 121s, attach disk operation succeeded and finally azure vm has attached one non-existing disk What you expected to happen: should not delete an azure disk when that disk is being attached How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: The difficulty here is when a disk is being attached or detached, the disk state is still "unattached", would be the reason why delete disk at that time would succeed: Azure disk RP would have two options to fix this issue: a new disk state: , disk delete operation when disk is being attached or detached Otherwise on azure provider(client) side, we cannot know whether that disk is being attached or detached since the second delete disk operation is done after 1 min Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: func (util *GCEDiskUtil) DetachDisk(pd *gcePersistentDisk) error { // Unmount the global PD mount, which should be the only one. globalPDPath := makeGlobalPDName(pd.plugin.host, pd.pdName) <ins> glog.V(5).Infof("DetachDisk(pd) where pd is %#v and the globalPDPath is %qrn", pd, globalPDPath) // Terminate any in progress verify detach go routines, this will block until the goroutine is ready to exit because the channel is unbuffered detachCleanupManager.Send(pd.pdName, true) </ins> if err := pd.mounter.Unmount(globalPDPath); err != nil { return err }

The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.


Question: "k8s.io/api/core/v1" rbacv1beta1 "k8s.io/api/rbac/v1beta1" storagev1 "k8s.io/api/storage/v1" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/sets"

Failing on gke only: https://k8s- https://k8s-
/assign
Unfortunately the exact error message that its failing with is lost due to a typo: I have fixed that typo locally and am running the tests repeatedly to try to reproduce the flake. have been unable to reproduce the flake on HEAD of Kubernetes in GKE in 15 runs so far. Running 150 tests overnight.
Unable to repro on my own GKE cluster. Merged the error message fix, will monitor for the next flake to debug more


Question: "os" "time" <del> kube_client "github.com/GoogleCloudPlatform/kubernetes/pkg/client" </del> <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/registry" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/coreos/go-etcd/etcd"

It would be great to have an easy way to run a kubernetes master/minion locally. Currently, it's not that easy to set up. We should have a convenient script to make it work.
The integration test binary is pretty close. It'd be pretty simple to glue in the kubelet to that binary. Brendan On Jun 10, 2014 2:53 PM, "Daniel Smith" wrote:
I'll take a look.
+1 for this -- especially in script form rather than a go executable.
what exactly do you mean by "script form"? Were you thinking of something like a version of cloudcfg that can be run without needing to have started up a daemon first?
Basically, what I had in mind, is a shell script that looks something like
Ah, ok. That's very similar to local-; the only difference is that everything is packaged into a single go binary. We probably will switch to using individual binaries for this when we start running our components inside docker containers.
The script just makes things more obvious how to run things for those who are looking to integrate them into non-GCE environments.
That's a good point. I'll look into this some more later.
Create a pad as follows: Name: non-persistent-redis Container image: redis Named-volum with name: cache-control Mount path: /data/redis It should launch in the pre-prod namespace and the volume MUST NOT be persistent. apiVersion: v1 kind: Pod metadata: name: non-persistent-redis namespace: pre-prod spec: containers: name: non-persistent-redis image: redis volumeMounts: name: cache-control mountPath: "/data/redis" volumes: name: cache-control emptyDir: {}

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.

When I'm accessing persistentVolumes and persistentVolumeClaims, I'm getting nil instead of empty list of items, as happening in other entities when there are no items. { "kind": "PersistentVolumeList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/persistentvolumes", "resourceVersion": "" } }
Yes, this seems like a bug. I see an empty list when I access pods: And items is nil when I access persistent volumes:
Items shouldn't be tagged with omitempty:
cc
PersistentVolumeList and PersistentVolumeClaimList appear to be the only resource types with this problem.
Please see for fix.


Question: ginkgo.By("Waiting for resource to become available on the local node after re-registration") gomega.Eventually(func() bool { <del> node := getLocalNode(f) return numberOfDevicesCapacity(node, resourceName) == devsLen && </del> <ins> node, ready := getLocalTestNode(f) return ready && numberOfDevicesCapacity(node, resourceName) == devsLen && </ins> numberOfDevicesAllocatable(node, resourceName) == devsLen }, 30*time.Second, framework.Poll).Should(gomega.BeTrue())

ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!

xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review


Question: # TODO: split this out into args when we no longer need to pipe stdout to a file #6428 - sh - -c <del> - '/glbc --verbose=true --default-backend-service=kube-system/default-http-backend --sync-period=600s --running-in-cluster=false --use-real-cloud=true --config-file-path=/etc/gce.conf --healthz-port=8086 1>>/var/log/glbc.log 2>&1' </del> <ins> - '/glbc --verbose=true --apiserver-host=http://localhost:8080 --default-backend-service=kube-system/default-http-backend --sync-period=600s --running-in-cluster=false --use-real-cloud=true --config-file-path=/etc/gce.conf --healthz-port=8086 1>>/var/log/glbc.log 2>&1' </ins> volumes: - hostPath: path: /etc/gce.conf

This leads to the following errors when running the example Guest Book application: If the proxy and the endpoint are on the same machine then the proxy works because everything defaults to 127.0.0.1 Successful proxy logs
I have this problem too. I think this has something to do with not having pod_registry/fillPodInfo called on the pods, but I'm not sure how to get the code where it needs to be.
I have repro-d this, I should be able to close it down soon. Sorry! --brendan , Ryan Fowler wrote:
please see for the fix. Sorry for the issue!
This issue is resolved. Closing.


Question: panic(err) } <ins> mismatchErrorMessage := "ERROR: %v indicates that %v should be at version %v, but the following files didn't match:nn" + "%vnnif you are changing the version of %v, make sure all of the following files are updated with the newest version including %vn" + "then run ./hack/verify-external-dependencies-version.shnn" </ins> externalDeps := &dependencies{} <ins> var pathsToUpdate []string </ins> err = yaml.Unmarshal(externalDepsFile, externalDeps) if err != nil { panic(err)

What would you like to be : The current error message is vague and could be improved. (credit to Why is this needed: Improve contributor experience, make it easier to identify change to be fixed. /cc
/assign


Question: "    VolumeHandle:t%vn"+ "    ReadOnly:t%vn", csi.Driver, csi.VolumeHandle, csi.ReadOnly) <ins> printCSIPersistentVolumeAttributesMultiline(w, "VolumeAttributes", csi.VolumeAttributes) } func printCSIPersistentVolumeAttributesMultiline(w PrefixWriter, title string, annotations map[string]string) { printCSIPersistentVolumeAttributesMultilineIndent(w, "", title, "t", annotations, sets.NewString()) } func printCSIPersistentVolumeAttributesMultilineIndent(w PrefixWriter, initialIndent, title, innerIndent string, attributes map[string]string, skip sets.String) { w.Write(LEVEL_2, "%s%s:%s", initialIndent, title, innerIndent) if len(attributes) == 0 { w.WriteLine("<none>") return } // to print labels in the sorted order keys := make([]string, 0, len(attributes)) for key := range attributes { if skip.Has(key) { continue } keys = append(keys, key) } if len(attributes) == 0 { w.WriteLine("<none>") return } sort.Strings(keys) for i, key := range keys { if i != 0 { w.Write(LEVEL_2, initialIndent) w.Write(LEVEL_2, innerIndent) } line := fmt.Sprintf("%s=%s", key, attributes[key]) if len(line) > maxAnnotationLen { w.Write(LEVEL_2, "%s...n", line[:maxAnnotationLen]) } else { w.Write(LEVEL_2, "%sn", line) } i++ } </ins> } type PersistentVolumeDescriber struct {


Question: "k8s.io/klog" ) <del> // NewSchedulerCommand creates a *cobra.Command object with default parameters func NewSchedulerCommand() *cobra.Command { </del> <ins> // Option configures a framework.Registry. type Option func(framework.Registry) error // NewSchedulerCommand creates a *cobra.Command object with default parameters and registryOptions func NewSchedulerCommand(registryOptions ...Option) *cobra.Command { </ins> opts, err := options.NewOptions() if err != nil { klog.Fatalf("unable to initialize command options: %v", err)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: By("deleting host0Pod") framework.ExpectNoError(podClient.Delete(host0Pod.Name, api.NewDeleteOptions(0)), "Failed to delete host0Pod") } <ins> By("Test completed successfully, waiting for PD to safely detach") waitForPDDetach(disk1Name, host0Name) waitForPDDetach(disk2Name, host0Name) </ins> }) })

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: fmt.Sprintf("sequence No: %v->%v", rc.Status.ObservedGeneration, newStatus.ObservedGeneration)) rc.Status = newStatus <del> _, updateErr = c.UpdateStatus(rc) if updateErr == nil || i >= statusUpdateRetries { return updateErr </del> <ins> updatedRC, updateErr = c.UpdateStatus(rc) if updateErr == nil { return updatedRC, nil } // Stop retrying if we exceed statusUpdateRetries - the replicationController will be requeued with a rate limit. if i >= statusUpdateRetries { break </ins> } // Update the controller with the latest resource version for the next poll if rc, getErr = c.Get(rc.Name, metav1.GetOptions{}); getErr != nil { // If the GET fails we can't trust status.Replicas anymore. This error // is bound to be more interesting than the update failure. <del> return getErr </del> <ins> return nil, getErr </ins> } } <ins> return nil, updateErr </ins> } // OverlappingControllers sorts a list of controllers by creation timestamp, using their names as a tie breaker.

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: package e2e import ( <ins> "io/ioutil" "net/http" </ins> "strings" "time"

xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


Question: {% if grains.cloud == 'azure' %} MACHINES="{{ salt['mine.get']('roles:kubernetes-pool', 'grains.items', expr_form='grain').values()|join(',', attribute='hostnamef') }}" {% set machines = "-machines $MACHINES" %} <del> {% elif grains.cloud is defined and grains.cloud == 'rackspace' %} MACHINES="{{ salt['mine.get']('roles:kubernetes-pool', 'grains.items', expr_form='grain').values()|map(attribute='ip_interfaces.eth2')|map('first')|join(',') }}" {% set machines = "-machines $MACHINES" %} {% set minion_regexp = "" %} </del> {% endif %} <del> {% if grains.cloud == 'vsphere' %} </del> <ins> {% if grains.cloud == 'rackspace' or grains.cloud == 'vsphere' %} </ins> # Collect IPs of minions as machines list. <del> # Use a bash array to build the value we need. It doesn't appear to be # possible call functions map or zip, or use lambda's from Jinja. </del> <ins> # # Use a bash array to build the value we need. Jinja 2.7 does support a 'map' # filter that would simplify this.  However, some installations (specifically # Debian Wheezy) only install Jinja 2.6. </ins> MACHINE_IPS=() {% for addrs in salt['mine.get']('roles:kubernetes-pool', 'network.ip_addrs', expr_form='grain').values() %} MACHINE_IPS+=( {{ addrs[0] }} ) {% endfor %} {% set machines = "-machines=$(echo ${MACHINE_IPS[@]} | xargs -n1 echo | paste -sd,)" %} <ins> {% set minion_regexp = "" %} </ins> {% endif %} {% endif %}

The Rackspace integration broke the GCE deploy. With Debian Wheezy the jinja2 version (package python-jinja2) is jinja2 version 2.6. This doesn't have support for the filter used in . Even though we don't hit that case for GCE, jinja apparently verifies the entire file. The VMWare folks hit this too and worked around it with a funky loop. We probably need to use the same logic for rackspace. (The jinja2 version comes from the debian repository instead of pypi as installed by the salt bootstrap scripts. We might be able to grab a newer version for GCE but that seems more difficult.)
I can get to this by EOD but if someone else wants to pick it up feel free.
I'll see if the VMWare code is workable for us.
I'm trying that out now. Thanks for following up.
My only issue with this solution for Rackspace is the use of eth0. For us, eth0 is a public IP address and means communication between apiserver <-kubelet will be over the net. It's fine for the time being but I'll send an updated PR that re-focuses on eth2 for Rackspace.
I missed that! I should have looked more carefully. I'll get this one in and we can fix it up and try and share some code.
Sounds good. I'll have some other minor updates as well.


Question: } if !initializing { restarts = 0 <ins> hasRunning := false </ins> for i := len(pod.Status.ContainerStatuses) - 1; i >= 0; i-- { container := pod.Status.ContainerStatuses[i]

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When one pod is running and has two Containers. Container 1 exits with success and container 2 is running. gives correct info - the pod status is . However, , the pod status is . What you expected to happen: , the pod status is . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
i can not see your pod named nginx in your above code
, updated the typo.
/sig node /remove-sig api-machinery


Question: "properties": { "strategy": { "default": "", <del> "description": "strategy specifies how custom resources are converted between versions. Allowed values are: - `None`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `Webhook`: API Server will call to an external webhook to do the conversion. Additional informationn  is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.", </del> <ins> "description": "strategy specifies how custom resources are converted between versions. Allowed values are: - `"None"`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `"Webhook"`: API Server will call to an external webhook to do the conversion. Additional informationn  is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.", </ins> "type": "string" }, "webhook": {

Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


Question: Port:     uint16(80), Protocol: string("TCP"), } <ins> rss := []*utilipvs.RealServer{ {net.ParseIP("172.16.2.1"), 8080, 1}, {net.ParseIP("172.16.2.2"), 8080, 2}, {net.ParseIP("172.16.2.3"), 8080, 3}, } </ins> err := fake.AddVirtualServer(vs) if err != nil { t.Errorf("Fail to add virutal server, error: %v", err) } <del> // Add a real server to the virtual server rs1 := &utilipvs.RealServer{ Address: net.ParseIP("172.16.2.1"), Port:    uint16(8080), Weight:  1, } err = fake.AddRealServer(vs, rs1) if err != nil { t.Errorf("Fail to add real server, error: %v", err) } // Add another real server to the virtual server rs2 := &utilipvs.RealServer{ Address: net.ParseIP("172.16.3.2"), Port:    uint16(8080), Weight:  2, } err = fake.AddRealServer(vs, rs2) if err != nil { t.Errorf("Fail to add real server, error: %v", err) } // List all real servers of the virtual server list, err := fake.GetRealServers(vs) if err != nil { t.Errorf("Fail to get real servers of the virtual server, error: %v", err) } if len(list) != 2 { t.Errorf("Expect 2 virutal servers, got: %d", len(list)) </del> <ins> // Add real server to the virtual server for i := range rss { if err = fake.AddRealServer(vs, rss[i]); err != nil { t.Errorf("Fail to add real server, error: %v", err) } </ins> } // Delete a real server of the virtual server <del> err = fake.DeleteRealServer(vs, rs2) list, err = fake.GetRealServers(vs) if err != nil { t.Errorf("Fail to get real servers of the virtual server, error: %v", err) </del> <ins> // Make sure any position of the list can be real deleted rssLen := len(rss) for i := range rss { // List all real servers of the virtual server list, err := fake.GetRealServers(vs) if err != nil { t.Errorf("Fail to get real servers of the virtual server, error: %v", err) } if len(list) != rssLen { t.Errorf("Expect %d virutal servers, got: %d", len(rss), len(list)) } rsToDel := list[i] if err = fake.DeleteRealServer(vs, rsToDel); err != nil { t.Errorf("Fail to delete real server of the virtual server, error: %v", err) } else { dests, err := fake.GetRealServers(vs) if err != nil { t.Errorf("Fail to get real servers of the virtual server, error: %v", err) } for _, dest := range dests { if toRealServerKey(dest).String() == toRealServerKey(rsToDel).String() { t.Errorf("Expect real server %q be deleted.", rsToDel.String()) } } if err = fake.AddRealServer(vs, rsToDel); err != nil { t.Errorf("Fail to add real server, error: %v", err) } } } // Test delete real server that not exist rs := &utilipvs.RealServer{ Address: net.ParseIP("172.16.2.4"), Port:    uint16(8080), Weight:  1, </ins> } <del> if len(list) != 1 { t.Errorf("Expect 1 real server, got: %d", len(list)) </del> <ins> if err = fake.DeleteRealServer(vs, rs); err == nil { t.Errorf("Delete real server that not exist, Expect error, got nil") </ins> } // Delete the virtual server err = fake.DeleteVirtualServer(vs)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS real server can't be right deleted in FakeIPVS for ipvs testing. What you expected to happen: When delete ipvs real server, we can't give the weight information, so we can't use deep equal in .
/sig network /area ipvs
/kind bug


Question: var netexecImageName = imageutils.GetE2EImage(imageutils.Agnhost) <ins> // TranslateIPv4ToIPv6 maps an IPv4 address into a valid IPv6 address // adding the well known prefix "0::ffff:" https://tools.ietf.org/html/rfc2765 // if the ip is IPv4 and the cluster IPFamily is IPv6, otherwise returns the same ip func TranslateIPv4ToIPv6(ip string) string { if TestContext.IPFamily == "ipv6" && !k8utilnet.IsIPv6String(ip) { ip = "0::ffff:" + ip } return ip } </ins> // NewNetworkingTestConfig creates and sets up a new test config helper. func NewNetworkingTestConfig(f *Framework) *NetworkingTestConfig { config := &NetworkingTestConfig{f: f, Namespace: f.Namespace.Name, HostNetwork: true}

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: } }) <del> // This test creates a load balancer, make sure its health check interval // equals to gceHcCheckIntervalSeconds. Then the interval is manipulated // to be something else, see if the interval will be reconciled. ginkgo.It("should reconcile LB health check interval [Slow][Serial][Disruptive]", func(ctx context.Context) { const gceHcCheckIntervalSeconds = int64(8) // This test is for clusters on GCE. // (It restarts kube-controller-manager, which we don't support on GKE) e2eskipper.SkipUnlessProviderIs("gce") e2eskipper.SkipUnlessSSHKeyPresent() clusterID, err := gce.GetClusterID(ctx, cs) if err != nil { framework.Failf("framework.GetClusterID(cs) = _, %v; want nil", err) } gceCloud, err := gce.GetGCECloud() if err != nil { framework.Failf("framework.GetGCECloud() = _, %v; want nil", err) } namespace := f.Namespace.Name serviceName := "lb-hc-int" jig := e2eservice.NewTestJig(cs, namespace, serviceName) ginkgo.By("create load balancer service") // Create loadbalancer service with source range from node[0] and podAccept svc, err := jig.CreateTCPService(ctx, func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer }) framework.ExpectNoError(err) ginkgo.DeferCleanup(func(ctx context.Context) { ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(ctx, cs, svc.Namespace, svc.Name) }) svc, err = jig.WaitForLoadBalancer(ctx, e2eservice.GetServiceLoadBalancerCreationTimeout(ctx, cs)) framework.ExpectNoError(err) hcName := gcecloud.MakeNodesHealthCheckName(clusterID) hc, err := gceCloud.GetHTTPHealthCheck(hcName) if err != nil { framework.Failf("gceCloud.GetHttpHealthCheck(%q) = _, %v; want nil", hcName, err) } gomega.Expect(hc.CheckIntervalSec).To(gomega.Equal(gceHcCheckIntervalSeconds)) ginkgo.By("modify the health check interval") hc.CheckIntervalSec = gceHcCheckIntervalSeconds - 1 if err = gceCloud.UpdateHTTPHealthCheck(hc); err != nil { framework.Failf("gcecloud.UpdateHttpHealthCheck(%#v) = %v; want nil", hc, err) } ginkgo.By("restart kube-controller-manager") if err := e2ekubesystem.RestartControllerManager(ctx); err != nil { framework.Failf("e2ekubesystem.RestartControllerManager() = %v; want nil", err) } if err := e2ekubesystem.WaitForControllerManagerUp(ctx); err != nil { framework.Failf("e2ekubesystem.WaitForControllerManagerUp() = %v; want nil", err) } ginkgo.By("health check should be reconciled") pollInterval := framework.Poll * 10 loadBalancerPropagationTimeout := e2eservice.GetServiceLoadBalancerPropagationTimeout(ctx, cs) if pollErr := wait.PollImmediate(pollInterval, loadBalancerPropagationTimeout, func() (bool, error) { hc, err := gceCloud.GetHTTPHealthCheck(hcName) if err != nil { framework.Logf("ginkgo.Failed to get HttpHealthCheck(%q): %v", hcName, err) return false, err } framework.Logf("hc.CheckIntervalSec = %v", hc.CheckIntervalSec) return hc.CheckIntervalSec == gceHcCheckIntervalSeconds, nil }); pollErr != nil { framework.Failf("Health check %q does not reconcile its check interval to %d.", hcName, gceHcCheckIntervalSeconds) } }) </del> // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP on [Slow] [LinuxOnly]", func(ctx context.Context) { // L4 load balancer affinity `ClientIP` is not supported on AWS ELB.

master-informing: ! 2023-09-07 03:13:59 +0000 UTC No response /sig network
Checking this failure This is running ingress-gce version, logs in failure is It seems the healthceck interval is not updated correctly, by reading the test tagging the people with more expertise in this area /cc
/cc
it seems the test restarts the kube-controller-manager but we landed recently the switch from intree to outtree cloud providers, so this logic now runs in the cloud-controller-manager, can this be the problem, that we have to restart the cloud-controller-manager now instead of the kcm?
/triage accepted This flake started after and kept failing after 09-07.
yeah, the logic that handles the loadbalancer shifted to that cloud controller manager with that PR


Question: } // ValidateEndpointsUpdate tests to make sure an endpoints update can be applied. <ins> // NodeName changes are allowed during update to accommodate the case where nodeIP or PodCIDR is reused. // An existing endpoint ip will have a different nodeName if this happens. </ins> func ValidateEndpointsUpdate(newEndpoints, oldEndpoints *core.Endpoints) field.ErrorList { allErrs := ValidateObjectMetaUpdate(&newEndpoints.ObjectMeta, &oldEndpoints.ObjectMeta, field.NewPath("metadata")) <del> allErrs = append(allErrs, validateEndpointSubsets(newEndpoints.Subsets, oldEndpoints.Subsets, field.NewPath("subsets"))...) </del> <ins> allErrs = append(allErrs, validateEndpointSubsets(newEndpoints.Subsets, field.NewPath("subsets"))...) </ins> allErrs = append(allErrs, ValidateEndpointsSpecificAnnotations(newEndpoints.Annotations, field.NewPath("annotations"))...) return allErrs }

/kind bug What happened: We have noticed that one of the node-exporters is down. It turned out that service has "old" endpoint still. After digging deeper i see following errors in controller manager Turns out controller manager can not update endpoints for node-exporter service, because of this error. In endpoint list is see right IPs for worker nodes, but wrong IP for master node (that's why we detected the problem). Context: two worker nodes were recreated with new hostnames (generated randomly), but they use same ip address one master node were recreated with new hostname and new ip I've tried to manually update hostname and got same error, seems nodeName is not updatable field. It also affects all other endpoints, we were forced to recreated all pods (so they will change IPs) to be able to fix endpoints reconciliation. What you expected to happen: controller manager successfully update endpoints after node recreated with same IP, but different hostname. How to reproduce it (as minimally and precisely as possible): 1 master, 1 worker, service that points to pods with host network. k8s API service (keep it shut down) worker node with same IP and different host name to recreate pod on worker and in controller manager should appear error above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): CoreOS 1688.5.3 Kernel (e.g. ): 4.14.32-coreos Install tools: own tool () Others:
/sig node
what is the use case of changing node name?
/sig api-machinery
We have ephemeral immutable nodes in our clusters. So upgrade or any other operation will cause node recreation. Hostname generated automatically, but the IP address depends on the underlay hypervisor node, where node will be scheduled. So it's possible that node stays on the same hypervisor and uses same ip address.
/cc
/assign
/cc do you think it's reasonable to loose endpoints validation?
/sig network this API is owned by sig network


Question: activeChains[svcChain] = true // Capture the clusterIP. <del> writeLine(rulesLines, </del> <ins> args := []string{ </ins> "-A", string(iptablesServicesChain), "-m", "comment", "--comment", fmt.Sprintf(""%s cluster IP"", name.String()), "-m", protocol, "-p", protocol, "-d", fmt.Sprintf("%s/32", info.clusterIP.String()), "--dport", fmt.Sprintf("%d", info.port), <del> "-j", string(svcChain)) </del> <ins> } if proxier.MasqueradeAll { writeLine(rulesLines, append(args, "-j", "MARK", "--set-xmark", fmt.Sprintf("%s/0xffffffff", iptablesMasqueradeMark))...) } writeLine(rulesLines, append(args, "-j", string(svcChain))...) </ins> // Capture externalIPs. for _, externalIP := range info.deprecatedPublicIPs {

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "NodeSelectorTerm") <del> ep.pattern = append(ep.pattern, "Storage node affinity check failed") </del> <ins> ep.pattern = append(ep.pattern, "MountVolume.NodeAffinity check failed") </ins> It("should not be able to mount due to different NodeName", func() { testPodWithNodeName(config, testVolType, ep, config.nodes[1].Name, makeLocalPodWithNodeName, testMode)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Failing since 12/17 https://k8s- /assign
Looks like the error message changed.
I managed to get e2e test running on my multi node cluster, and it looks ok to me. All tests are passing, here is the log I see: [sig-storage] PersistentVolumes-local [Feature:LocalPersistentVolumes] [Serial] [Volume type: tmpfs] when pod's node is different from PV's NodeName should not be able to mount due to different NodeName [BeforeEach] [sig-storage] PersistentVolumes-local [Feature:LocalPersistentVolumes] [Serial] STEP: Creating a kubernetes client Dec 29 13:55:55.865: INFO: kubeConfig: STEP: Building a namespace api object STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-storage] PersistentVolumes-local [Feature:LocalPersistentVolumes] [Serial] [BeforeEach] [Volume type: tmpfs] [BeforeEach] when pod's node is different from PV's NodeName [It] should not be able to mount due to different NodeName ResourceVersion: FieldPath:} Reason:FailedMount Message:Storage node affinity check failed for volume "local-pv8v7k7" : NodeSelectorTerm [{ Operator:In Values:[cluster-02-]}] does not match node labels Source:{Component:kubelet Host:cluster-02-} Maybe the issue you see is gce specific? Could you please add this for loop after this line? for _,e := range events.Items { fmt.Printf("<<<<%+vn", e) } And send me logs I would like to compare with logs I see on my cluster.
are you working off of the latest k8s code? There was a change that went in mid-Dec that changed a bunch of volume error messages
yep on the latest master


Question: // Watch implements storage.Interface. func (c *Cacher) Watch(ctx context.Context, key string, opts storage.ListOptions) (watch.Interface, error) { pred := opts.Predicate <ins> // If the resourceVersion is unset, ensure that the rv // from which the watch is being served, is the latest // one. "latest" is ensured by serving the watch from // the underlying storage. if opts.ResourceVersion == "" { return c.storage.Watch(ctx, key, opts) } </ins> watchRV, err := c.versioner.ParseResourceVersion(opts.ResourceVersion) if err != nil { return nil, err

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: // inter-pod affinity for pods of different rollouts and anti-affinity // for pods of the same rollout, so it will need to be updated when // performing a rollout. <del> setAffinity(d) </del> <ins> setAffinities(d, false) </ins> d.Spec.Strategy.RollingUpdate = &appsv1.RollingUpdateDeployment{ MaxSurge:       intOrStrP(1), MaxUnavailable: intOrStrP(0),

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:


Question: <ins> apiVersion: v1 kind: Service metadata: name: a spec: selector: app: test clusterIP: None ports: - port: 80 </ins>

/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign


Question: numErrs: 0, }, { <del> name: "invalid port in use", </del> <ins> name: "external port in use", </ins> svc: api.Service{ ObjectMeta: api.ObjectMeta{Name: "abc123", Namespace: api.NamespaceDefault}, Spec: api.ServiceSpec{

From a user: I'm trying to expose multiple services on the same port in a GKE Kubernetes 0.8.1 cluster. My question is similar to this SO post but it is not solved by 0.5.x+ I've deployed a service (hellogo) with an external load balancer which works correctly. Deploying another service (pynote) which is extremely similar fails { "apiVersion": "v1beta1", "kind": "Service", "id": "pynote-service", "selector": { "name": "pynote" }, "containerPort": 8080, "protocol": "TCP", "port": 80, "createExternalLoadBalancer": true } gcloud preview container services --cluster athena --zone $ZONE create --config-file pynote- ERROR: F0124 19:42:27. ] service "pynote-service" is invalid: service "pynote-service" cannot be updated: port: 80 is already in use If I delete the other service and start pynote first, the service and load balancer are setup correctly, but the other service cannot be deployed with the same error. I believe the current proxying design is that service routing occurs via proxies on every node which are part of the target pool. Every service gets its own IP. Putting up two external services on port 80 on GKE should allocate two load balancers, each with a fixed IP address. This design does not preclude exposing services on the same port as each service is accessed through a separate external IP. Furthermore, Kubernetes itself runs a service (kubernetes-ro) on port 80 which coexists with my first service nicely. At the very least, the message is misleading. There is no requirement that the services deployed on a cluster use unique ports. In the source, it becomes apparent that its the external load balancer creation that causes the command to fail. Indeed, I can deploy pynote with createExternalLoadBalancer to false.

Happily coexisting: hellogo-service <nonename=hellogo 10.143.249.126 80 pynote-service <nonename=pynote 10.143.254.255 80 I've tried to define my own target pool called pynote-service, selecting all the cluster node instances and created a network load balancer with a forwarding rule on port 80 to the target pool. Despite the fact that it corresponds to the one created automatically for hellogo-service, it does not seem to resolve to the running pynote pod. This feels like I'm missing something small, but I've been blocked on this. Perhaps I'll only ever run one lonely web service...
I believe there's a bug in the service validation logic, left over from when you actually might have a conflict...


Question: "net/http/httptest" "net/url" "reflect" <del> "strings" </del> <ins> "regexp" </ins> "testing" "k8s.io/apimachinery/pkg/util/diff"

unit tests since switching to go1.18 No response No response Weirdly, the test is passing in CI () and failing locally /sig api-machinery auth testing
seems related Jordan, are you using mac ?
yes, though the description in that issue seems to be not just test-only changes to the errors received from invalid certificates (which is what I saw), but actual certificates that previously validated
marking as release blocker until it is determined that go1.18 is not rejecting valid certificates
this needs an assignee and a reproducer we can file as an upstream issue for golang to ask why previously valid certificates now fail validation
marking critical until root-caused and determined whether this is prod-impacting, since (in my opinion) this should block rc0
On linux it works fine: on MacOS, it fails:
Looks like the change in golang that broke the scenario was:
if the only impact is a different failure message on an already-invalid certificate, I don't care and we can update the test to accept either message is what makes me wonder what previously-accepted certificates are now failing
reproducer test case here. this one has no issues under go 1.17 and fails with go 1.18 (stripped down version of ) : :
the new error message is minted here: i suspect it is related to: So essentially this seems to be something that may hit in our case. right?
yeah, if this is scoped to just darwin clients, and only ones using system roots, that greatly narrows the impact
Asked on gophers slack -
xref:
opened to tolerate the additional errors discussion in indicates the tightened validation is very bounded (darwin-only, so clients-only, system roots only), and certificates issued by public CAs would be expected to pass this validation dropping release-blocking label and priority to important-soon, but should still make it into 1.24 to make unit tests pass locally


Question: <del> # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM busybox ADD cauldron cauldron ADD cauldron.go cauldron.go ENTRYPOINT ["/cauldron"] </del>

We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) Apparently the repo is referenced for docker push and pull in It's unclear to me whether we have broken anything, or if this is dead code. I would use to find out if this is referenced anywhere (aside from sundry vendor dirs) If dead code: remove it If used: move to test/e2e/images and follow the same pattern there used to push to /sig testing /area test /wg k8s-infra /priority important-soon /help /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) /area test /wg k8s-infra /priority important-soon /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, nil } <ins> // Name returns extenderURL to identifies the extender. func (h *HTTPExtender) Name() string { return h.extenderURL } </ins> // IsIgnorable returns true indicates scheduling should not fail when this extender // is unavailable func (h *HTTPExtender) IsIgnorable() bool {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In pkg/scheduler/core/generic_scheduler# , add some debug info: each scheduler extender's score and its name for each pod, e.g. And we need add a function(e.g. ) in interface to get extender's name or similar identity. For , it might be enough to return its url . Why is this needed: If there are two or more scheduler extenders configured for scheduler(and we have now), it will be very useful to debug why kubelet nodes get so high/low score. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Why don't you send a PR instead?
Thanks. PR is sent. PTAL.


Question: // Create the subPath file on the host existing := path.Join(source.Path, subPath) <del> externalIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) </del> <ins> nodeIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) if err != nil { nodeIP, err = framework.GetNodeInternalIP(&nodeList.Items[0]) } </ins> framework.ExpectNoError(err) <del> result, err := framework.SSH(fmt.Sprintf("echo "mount-tester new file" > %s", existing), externalIP, framework.TestContext.Provider) </del> <ins> result, err := framework.SSH(fmt.Sprintf("echo "mount-tester new file" > %s", existing), nodeIP, framework.TestContext.Provider) </ins> framework.LogSSHResult(result) framework.ExpectNoError(err) if result.Code != 0 {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: "[sig-storage] HostPath [It] should support existing single file subPath" test failed on environments which don't support external IPs like What you expected to happen: The test can pass successfully. How to reproduce it (as minimally and precisely as possible): just kick: Environment: Kubernetes version (use ): v1.11.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig storage