Question: t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} <del> sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </del> <ins> sgList, setupSg, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) </ins> assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) <ins> assert.False(t, setupSg, "Security Groups Setup Permissions Flag expected=%t , returned=%t", false, setupSg) </ins> }) } }

What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).

All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: } } <ins> // Returns a volumeAttachment object func NewVolumeAttachment(vaName, pvName, nodeName string, status bool) *storagev1.VolumeAttachment { return &storagev1.VolumeAttachment{ ObjectMeta: metav1.ObjectMeta{ UID:  types.UID(vaName), Name: vaName, }, Spec: storagev1.VolumeAttachmentSpec{ Attacher: "test.storage.gke.io", NodeName: nodeName, Source: storagev1.VolumeAttachmentSource{ PersistentVolumeName: &pvName, }, }, Status: storagev1.VolumeAttachmentStatus{ Attached: status, }, } } // Returns a persistentVolume object func NewPV(pvName, volumeName string) *v1.PersistentVolume { return &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ UID:  types.UID(pvName), Name: pvName, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: volumeName, }, }, }, } } func attachVolumeToNode(volumeName, nodeName string) { // if nodeName exists, get the object.. if not create node object var node *v1.Node found := false nodes.Size() for i := range nodes.Items { curNode := nodes.Items[i] if curNode.ObjectMeta.Name == nodeName { node = &curNode found = true break } } if !found { node = &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: nodeName, Labels: map[string]string{ "name": nodeName, }, Annotations: map[string]string{ util.ControllerManagedAttachAnnotation: "true", }, }, Status: v1.NodeStatus{ VolumesAttached: []v1.AttachedVolume{ { Name:       v1.UniqueVolumeName(TestPluginName + "/" + volumeName), DevicePath: "fake/path", }, }, }, } } else { volumeAttached := v1.AttachedVolume{ Name:       v1.UniqueVolumeName(TestPluginName + "/" + volumeName), DevicePath: "fake/path", } node.Status.VolumesAttached = append(node.Status.VolumesAttached, volumeAttached) } nodes.Items = append(nodes.Items, *node) } </ins> type TestPlugin struct { ErrorEncountered  bool attachedVolumeMap map[string][]string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Background: The attach/detach controller create objects for CSI Volumes to indicate that a volume needs to be attached to a node. It deletes the object to indicate that the volume should be detached. Problem: If the Attach/Detach controller crashes, and in the meantime the pod using that volume is deleted, when the AD controller restarts, it won't know that the volume was previously attached and should be cleaned up. What you expected to happen: Proposal: We should have the AD controller periodically list all objects and for any that are not accounted for, update its internal actual/desired state caches so that the volume is gracefully detached. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
could I take it?
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen /reopen
Reopened this issue. <detailsIn response to : /lifecycle frozen /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should address this


Question: var k8sBinDir = flag.String("k8s-bin-dir", "", "Directory containing k8s kubelet and kube-apiserver binaries.") <ins> var buildTargets = []string{ "cmd/kubelet", "cmd/kube-apiserver", "test/e2e_node/e2e_node.test", "vendor/github.com/onsi/ginkgo/ginkgo", } </ins> func buildGo() { glog.Infof("Building k8s binaries...") k8sRoot, err := getK8sRootDir() if err != nil { glog.Fatalf("Failed to locate kubernetes root directory %v.", err) } <del> cmd := exec.Command(filepath.Join(k8sRoot, "hack/build-go.sh")) </del> <ins> cmd := exec.Command(filepath.Join(k8sRoot, "hack/build-go.sh"), buildTargets...) </ins> cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr err = cmd.Run()

All the LoadBalancers tests It seems that for a long time IPVS and iptable implementations differ, it may be possible the test do some wrong assumptions? At this point is up to the IPVS maintainers if they want to run those tests or not, is ok if they don't have time to investigate to disable the tests, but we should not have flaky test running or we'll have a lot of noise /sig network /area ipvs /help
/sig network /area ipvs /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig network /area ipvs /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: defaultArguments["etcd-cafile"] = filepath.Join(cfg.CertificatesDir, kubeadmconstants.EtcdCACertName) defaultArguments["etcd-certfile"] = filepath.Join(cfg.CertificatesDir, kubeadmconstants.APIServerEtcdClientCertName) defaultArguments["etcd-keyfile"] = filepath.Join(cfg.CertificatesDir, kubeadmconstants.APIServerEtcdClientKeyName) <ins> // Apply user configurations for local etcd if cfg.Etcd.Local != nil { if value, ok := cfg.Etcd.Local.ExtraArgs["listen-client-urls"]; ok { defaultArguments["etcd-servers"] = value } } </ins> } if features.Enabled(cfg.FeatureGates, features.HighAvailability) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create a cluster, the specified parameter "listen-client-urls" does not work. kubeadm init --config=kube- fail with connection refuse for get http://127.0.0.1:2379 I think the element 'listen-client-urls' is unrecognized if the customized value is not equal '127.0.0.1' Here is my kube- No matter how I set the "listen-client-urls" parameter in the kube- , it is awlays be "https://127.0.0.1:2379" What you expected to happen: The parameter "listen-client-urls" can work! How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux kube-dev 3.10.0-862.el7.x8664 SMP Fri Apr 20 16:44:24 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: Others:
/sig cluster-lifecycle
/lifecycle active
kubeadm 1.13 has set the default value to Default route transits interface ip address instead of loopback ip


Question: # - centos 7 # - debian jessie <del> # RHEL os detection cat /etc/*-release | grep "ID="rhel"" OS_RHEL=$? </del> # On a systemd environment, enable cpu and memory accounting for all processes by default. if [ -d /etc/systemd ]; then cat <<EOF >kubernetes-accounting.conf

in e2enode test curls and directly pipes to shell: The issue is that this may present a supply chain security risk, as we are directly executing the script from I would instead expect one of the following: host includes docker already via it's OS image script is checked into this repo and version controlled script is at least verified to be unchanged via hash Running anything from - , such as: No response All GCP (kind of) N/A N/A N/A N/A
/sig node
that's some pretty old bash, I don't think we're actually using this in CI, probably that codepath can just be dropped, but up to the SIG node folks.
/triage accepted /priority backlog /assign
the script is being referred from the docs: I think the safest bet is to simply remove the docker installation part.


Question: } return schema.GroupVersion{} } <del> func hasZeroLimitedNominalConcurrencyShares(obj *flowcontrol.PriorityLevelConfiguration) bool { return obj != nil && obj.Spec.Limited != nil && obj.Spec.Limited.NominalConcurrencyShares == 0 } </del>

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: import ( "fmt" <ins> "sort" "github.com/golang/glog" </ins> cadvisorapi "github.com/google/cadvisor/info/v1" "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CPU Manager uses topology from cAdvisor which uses which is different from in terms of coreID's. assigns coreID's in unique manner (platform-wise) whereas has coreID's from CPU registers which are unique only on socket level. This causes some unexpected cpu assignment on platforms with more than one socket. All tests written all "false positive" Example output: From What you expected to happen: Fix either by changing the way CPU manger sees topology (use ) or redesign CPU Manager to handle coreID's as socket only unique How to reproduce it (as minimally and precisely as possible): kubelet on dual socket node G pod that requests 2 cores pod gets 2 cores on different sockets
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: package printers import ( <ins> "encoding/base64" </ins> "encoding/json" "fmt" "io"

Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e.

That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function.

It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


Question: if err != nil { return err } <del> obj, err := cmdutil.UpdateObject(info, func(obj runtime.Object) error { err := labelFunc(obj, overwrite, resourceVersion, lbls, remove) </del> <ins> var outputObj runtime.Object if cmdutil.GetFlagBool(cmd, "dry-run") { err = labelFunc(info.Object, overwrite, resourceVersion, lbls, remove) if err != nil { return err } outputObj = info.Object } else { outputObj, err = cmdutil.UpdateObject(info, func(obj runtime.Object) error { err := labelFunc(obj, overwrite, resourceVersion, lbls, remove) if err != nil { return err } return nil }) </ins> if err != nil { return err } <del> return nil }) if err != nil { return err </del> } outputFormat := cmdutil.GetFlagString(cmd, "output") if outputFormat == "" { cmdutil.PrintSuccess(mapper, false, out, info.Mapping.Resource, info.Name, "labeled") } else { <del> f.PrintObject(cmd, obj, out) </del> <ins> return f.PrintObject(cmd, outputObj, out) </ins> } return nil })

kubectl should support a --dry-run option for every mutation to show what it would do, without actually doing it We might need API changes in order to make this high-fidelity. For instance, if we wanted to know what a pod config would look like after admission control populated defaults, we'd need to execute most of the creation path.
For generator types commands, I think there is a usability argument that the default when a user specifies should be to dry-run (and a user should be allowed to override that with ). I think I'd rather encourage composition of commands (pipe) than every generator command in the system creating objects. We may also want to clarify dry-run against the server vs dry run against the client - there are lots of places where we want to generate locally to a file or an editor or a pipe command before we want to create it. At this point, OpenShift has a few commands that treat -o  as "dry-run" - new-app (complex generator) and export (never expect to send it to the server). We also have a few that are designed to be compositional in a pipe always - process (take a template and turn it into a list of api objects), which never creates the objects for you (you have to pipe to ). I'm somewhat partial to having and support the presence of changing the default of to true, because that's more common and easier than having to set and . The counter argument is that is the preferred action, and is format only - except that by definition --dry-run using the default output format isn't very useful (you created a service named foo and an rc named bar). My gut is that most users once they understand what a particular generator does their default path is always going to be to review the change, and then if they want to create it they can pipe to or just by appending to their existing command. I'm not sure that's a strong enough argument to couple --dry-run and -o, but based on our experience so far it's going to be the "normal" path.
:+1: for this feature. In particular, I would like an ability to validate an object definition separate from actually creating the object. e.g. running at the validation step of my deployment pipeline.
I agree that the validation use case is important. I would distinguish --dry-run from --local.

The former would still entail contacting the server and returning the result of the mutation. The latter would imply that only local information were used.
- I'm planning to add a parameter like 'pretty' or 'labelselector' to API requests. Once the server get this parameter, it will do validation to the resource and check if request can pass admission controls, but return success before write to etcd. I've read the documented about API changing, it lists some principles when changing the api object, but parameters are not mentioned. Is there anything need to be taken special care of when adding a parameter?
We need to be careful about dry-run with admission control... some admission controllers (like quota, specifically), record things to etcd in the process of admitting an object. Also, dry-run wouldn't work with a list of objects intended to be created in order, where later objects reference earlier objects (and those references are validated at create time)
We could potentially carry "dry run" down into the admission controller request path. However, a query parameter may not be appropriate, instead there's probably a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. , Jordan Liggitt wrote:
- Thanks for advices! I'm trying to make myself thoroughly familar with request handling process. And during this period, I may need more of your help. I think I've replied this message before but unfortunately it's not showed up here:-( - Can you provide a link to the code about the HTTP header? Thanks!
This feels like a variant of an header with a sub parameter - possibly or possibly content-type . , Victor HU wrote:
+1 for this. My use case is to understand the difference between a stack file and an existing cluster, but this command would be a substitute.
It looks like support for has been . I think this issue can be closed.
We still need to make touch the server, or at least decide that we don't want to.
you mean move server side?
has just released kubeval: From to sig-apps mailing list: I am not sure if it validates the values and other constraints checked on the server-side. Just leaving it here as it might be relevant to the feature discussion. (h/t briangrant for pointing this out).
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. Do you remember why you said that? I'm trying to understand ...
should fix this issue, we can close it when the server-side dry-run feature reaches beta. The feature above () will fix this use-case, and even validate against the admission chain. is being built on top of server-side dry-run, and will be a good way to visualize the output of dry-run apply:
This is now beta.


Question: reason = pod.Status.Reason } <del> // If the Pod carries {type:PodScheduled, reason:WaitingForGates}, set reason to 'SchedulingGated'. </del> <ins> // If the Pod carries {type:PodScheduled, reason:SchedulingGated}, set reason to 'SchedulingGated'. </ins> for _, condition := range pod.Status.Conditions { if condition.Type == api.PodScheduled && condition.Reason == apiv1.PodReasonSchedulingGated { reason = apiv1.PodReasonSchedulingGated

<img width="844" alt="image" src="" <img width="942" alt="image" src="" Here are some incorrect comments and function name about SchedulingGated Comments and function names need to be changed to SchedulingGated nothing No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
/remove-kind bug /kind cleanup


Question: return probe.Failure, err.Error(), nil } req.Header = headers <ins> if headers.Get("Host") != "" { req.Host = headers.Get("Host") } </ins> res, err := client.Do(req) if err != nil { // Convert errors into failures to catch timeouts.

I noticed that httpHeaders were to the httpGet probe in to Sadly the httpHeaders addition to the httpGet probe does function correctly for all headers except the header. I think this must be to do with the way make a special case of the Host header . I have some code I used to debug the issue that I can share if needed.
Sigh, letting you set Host was the entire point of providing that option.
Does the host header work in probes now? I have been browsing the issues but can not tell, as I am having trouble with it in my setup.
my fix should be in kubernetes 1.2.3+ I found some code that exercises the issue... could you try this on your cluster I cannot reproduce in the environment I have to hand that is running k8s 1.7


Question: return probe.Failure, err.Error(), nil } req.Header = headers <ins> if headers.Get("Host") != "" { req.Host = headers.Get("Host") } </ins> res, err := client.Do(req) if err != nil { // Convert errors into failures to catch timeouts.

I noticed that httpHeaders were to the httpGet probe in to Sadly the httpHeaders addition to the httpGet probe does function correctly for all headers except the header. I think this must be to do with the way make a special case of the Host header . I have some code I used to debug the issue that I can share if needed.
Sigh, letting you set Host was the entire point of providing that option.
Does the host header work in probes now? I have been browsing the issues but can not tell, as I am having trouble with it in my setup.
my fix should be in kubernetes 1.2.3+ I found some code that exercises the issue... could you try this on your cluster I cannot reproduce in the environment I have to hand that is running k8s 1.7


Question: ginkgo.By("Waiting for devices to become unavailable on the local node") gomega.Eventually(func() bool { <del> return numberOfSampleResources(getLocalNode(f)) <= 0 </del> <ins> node, ready := getLocalTestNode(f) return ready && numberOfSampleResources(node) <= 0 </ins> }, 5*time.Minute, framework.Poll).Should(gomega.BeTrue()) <ins> ginkgo.By("devices now unavailable on the local node") </ins> }) ginkgo.It("Can schedule a pod that requires a device", func() {

ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!

xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review


Question: } } } <ins> func TestDoUnmountMountPoint(t *testing.T) { tmpDir1, err1 := utiltesting.MkTmpdir("umount_test1") if err1 != nil { t.Fatalf("error creating temp dir: %v", err1) } defer os.RemoveAll(tmpDir1) tmpDir2, err2 := utiltesting.MkTmpdir("umount_test2") if err2 != nil { t.Fatalf("error creating temp dir: %v", err2) } defer os.RemoveAll(tmpDir2) // Second part: want no error tests := []struct { mountPath    string corruptedMnt bool }{ { mountPath:    tmpDir1, corruptedMnt: true, }, { mountPath:    tmpDir2, corruptedMnt: false, }, } fake := &mount.FakeMounter{} for _, tt := range tests { err := doUnmountMountPoint(tt.mountPath, fake, false, tt.corruptedMnt) if err != nil { t.Errorf("err Expected nil, but got: %v", err) } } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: for nsn, count := range newEndpoints { if hcs.services[nsn] == nil { <del> klog.V(3).InfoS("Not saving endpoints for unknown healthcheck", "service", nsn) </del> continue } klog.V(3).InfoS("Reporting endpoints for healthcheck", "endpointCount", count, "service", nsn)

We are using Azure AKS V1.20.7 and the service type in production environment is ClusterIP. In Kube-proxy logs, continuously I see the below printed line. "Not saving endpoints for unknown healthcheck" ] Not saving endpoints for unknown healthcheck It may increase Syncing time I believe. Kindly assist. It should not print below line in kube-proxy logs. "Not saving endpoints for unknown healthcheck" ] Not saving endpoints for unknown healthcheck Service is of type - ClusterIP Validate kube-proxy logs in kube-system No response Azure AKS 1.20.7 Azure AKS <details</details<details</details<details</details<details</details
/sig network
/assign
/cc
I think that this is a duplicate of It also has a PR to fix it
/triage accepted I think we should remove that log line, is not meaningful
Just confirming, should I make a PR removing log line.
Thank you It won’t print that line in next k8s version ? Like 1.21.7? Any specific version?
I have doubts that this qualifies for backporting, because it is only happening if the log level verbosity is 3 or higher, a verbosity level of 2 use to be the default IIRC. Also, It's not impacting any functionality. Not backporting this change means that this only be present in current version 1.24, for previous versions people can configure the kube-proxy loglevel verbosity to 2 or lower, and it will not log this messge. is there any way users can define the kube-proxy verbosity level to 2? any inconvenient for doing that?
For AKS (managed cluster), I just checked and the default log verbosity configured is 3 to enable debug logging. For self-managed Kubernetes cluster, I think users have control over the log verbosity but for managed by default verbose logging is enabled? Have we backported any similar logging changes before?
WDYT about backporting this patch? ^^^
looks like is a log part, sgtm to backport
would you like to take care of the backports?
Do you want to open the PRs for backport? If not, lmk and I can open them.
Open PRs for backport:


Question: // If the pod network size is /113 or larger, the node CIDR will be set to the same // size and this will be rejected later in validation. // <del> // NOTE: Currently, the pod network must be /66 or larger. It is not reflected here, // but a smaller value will fail later validation. // </del> // NOTE: Currently, the design allows a maximum of 64K nodes. This algorithm splits // the available bits to maximize the number used for nodes, but still have the node // CIDR be a multiple of eight.

The comments in mention the IPv6 /66 restriction. This restriction was removed in PR Also, the unit tests show the /66 as the largest subnet. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/area ipv6


Question: name = "go_default_test", srcs = [ "admission_test.go", <ins> "broken_webhook_test.go", </ins> "main_test.go", ], tags = [

Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Message:            "currentMessage", }, currentNominatedNodeName: "node1", <del> expectedPatchRequests:    0, </del> <ins> expectedPatchRequests:    1, </ins> }, { name: "Should make patch request if pod condition already exists and is identical but nominated node name is set and different",

What happened: Scheduler decided that it wants to schedule Pod P on Node N after preempting some pods from N. It set the P.nominatedNodeName to N. After a while (before scheduling actually happened) node N was removed from the cluster. Scheduler did not reset P.nominatedNodeName to empty value. What you expected to happen: I would expect scheduler to unset P.nominatedNodeName when the node is no longer valid. The trivial reason for node not being valid any more is it is removed from the cluster. But there are more cases to cover. Out of the top of my head I may think of: extra Pod was scheduled to node and there is not enough resources any more either scheduled by scheduler or directly as static pod/daemonset node became unschedulable How to reproduce it (as minimally and precisely as possible): We did not try to reproduce issue manually. Yet following setup should work. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources. add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster Anything else we need to know?: Current behavior does not play well with Cluster Autoscaler which assumes that Pod with nominatedNodeName will be scheduled in a while. Therefore such pods do not trigger scale-up. If we can have P.nominatedNodeName set incorrectly for unbounded period of time CA will never provision node for P to run. Environment: Kubernetes version (use ): We observed the behavior during presubmit tests which run k8s built from HEAD of master on 27.11.2019.

Versions logged in test log: I1125 18:55:11.691] Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} I1125 18:55:11.692] Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE /sig scheduling
/assign
/cc
Another interesting case would be creating a new pod with pod antiaffinity blocking the pod with nominated node name from scheduling: up a cluster with pods, including a long-terminating low-priority pod A. high-priority pod B that has the same requests as A. pod C with negligible resource requests and zonal topology pod antiaffinity matching pod B (and no other pods in the cluster). What I'd expect: Pod C keeps pending, nominatedNodeName is cleared. Pod A is restarted on the same node it was running before. I haven't actually tested this scenario, so I'm not sure what will actually happen. If report is correct, this may be a more convenient way of reproducing the issue (no need to add / remove nodes, it's not important which pod will go to which node).
I'm not sure if we need to clear nominatedNodeName on Pod when node is no longer valid immediately. And we have other logic to clear nominatedNodeName. Reuse your example. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources.

add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster After this, The H will be scheduled again. Obviously it's going to fail. H will enter the preemption process and the is empty. It will return itself as preemption process will clean nominatedNodeName on the Pod H.
when I have some time I will check if reproduction steps I describe actually work. But we surely observed pods with lingering nominatedNodeName which pointed to inexistent node, during our E2E tests. Quick thought. The method has many return points. If it returned at the place you linked it seems should be cleared. But there are many return points below, which set the list of pods to be cleared to nil. Is it possible that for a pod to consistently exit from with one of those return points?
I think you are right. Actually there are many return points to return nil. I think maybe we need to clear the of the pod after the filter failed again and before the new preemption process to avoid potential risks. WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle stale
I believe it's a bug as in old logic, e.g., v1.18.3, if the Pod (with nominatedNodeName set) cannot be scheduled and also preemption doesn't help ( nil potentialNodes is a corner case, it's more common the pod can't fit on all potentialNodes), it won't hit L413: then in old recordingFailure logic, nominatedNodeName remains as is. In the latest codebase, introduced a fix to combine setting nominatedNodeName as well update the pod's unschedulable condition. However, this doesn't help in this issue either as it ignores the case when the pod's nominatedNodeName is nil:
/reopen as regresses - we need to come up with a complete fix.
Reopened this issue. <detailsIn response to : /reopen as regresses - we need to come up with a complete fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
what are the implications of not clearing the nominated node name?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: break } } <del> glog.V(2).Infof("at least one owner of object %s has FinalizerDeletingDependents, and the object itself has dependents, so it is going to be deleted with Foreground", item.identity) </del> <ins> glog.V(2).Infof("at least one owner of object %s has FinalizerDeletingDependents, and the object itself has dependents, so it is going to be deleted in Foreground", item.identity) </ins> // the deletion event will be observed by the graphBuilder, so the item // will be processed again in processDeletingDependentsItem. If it // doesn't have dependents, the function will remove the

Seeing this in v1.6.1. a Deployment and a Service the Deployment add an ownerRef that points back to the Service delete the Service The Service and the Deployment are deleted, but the ReplicaSet and pods remain. This is neither Orphan nor Foreground/Background policy. This behavior should never happen. Note: if I make a delete call to the API with propagationPolicy set explicitly then things behave as expected, so this only occurs when the resource's default propagation policy is being used.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
cc
This is expected. The default garbage collection policy for old controller types (rc, rs, deployment) etc. is orphaning, for backward compatibility. And when doing BackgroundGC, the GC controller doesn't set the deleteOptions.propagationPolicy. We met a similar problem for deleting deployment -RS -pods. Because that is a very common use case, we made a special fix in As you noticed, if you set deleteOptions.propagationPolicy to Orphan or ForegroundGC, it works as you expected.
that seems wrong. if I'm using the garbage collection feature to propagate (whether a user specified it or it was a default of a parent resource), I expect GC to set propagation policy.
rather than a one-off fix in the RS controller, why not address the break in the propagation chain in the GC controller?
In the backgroundGC, the owner is deleted immediately, so unlike orphaning or the foregroundGC, there is no finalizer left for the GC controller to determine the intention of the user. In the absence of the user's intention, the GC controller doesn't set deleteOptions.PropagationPolicy when deleting dependents.
If it's not orphaning, and it's not foregroundGC, the only logical option that remains is backgroundGC
There is another case, user might intentionally choose not to set DeleteOptions.Propagation policy and wants to respect the existing finalizers of the dependents or the default GCstrategy defined by the REST storage of the dependents.

except that it isn't... if I delete resource X and specify no propagation policy, that resource strategy's default propagation policy should take effect just as if the user had specified it.
Add CronJobs to the list of affected resources, since you have CronJob-Job-Pod chain. :+1: to what Jordan said just above.

BUG REPORT: <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version: 1.6.1 Environment: Hardware configuration: 2 identical machines, with i7-2600K CPUs and 4GB RAM. OS: CentOS 7 Kernel: 3.10.0-514.10.2.el7.x86_64 Install tools: KubeADM What happened: Deleting a deployment via the API with the DeleteOptions settings or deletes the Deployment and the replica set but NOT the pods. Deleting it with the DeleteOptions setting works as intended. What you expected to happen: The replica set deletion should propagate to the pods. I've read that the garbage collection should work for Deployments in 1.6. How to reproduce it: Anything else we need to know: I am using the and at the moment it only supports the orphanDependants option, since it's still at api version 1.5.
cc:
We discussed the problem in I'll change the GC to make sure it's cascading.
Sent
Any progress on this?
Still trying to figure out a sane fix:
I've encountered the same issue. My pods are still running even though I deleted the deployment.
lets come up with a fix for 1.7. This should work from the first time we introduced cascading deletion in Deployments.
Yes, we should. Do you have any comments on
replied. Let's move this forward. If the fix is small, I am going to backport it to 1.6.
Offline discussion with we punt this to v1.8.
Discussed with if we can get the PR fixed soon, it can still go in 1.7. It's a real bug fix.
Yes, we need to fix this for 1.7 and also backport to 1.6.

Should this be targeting 1.7?
yes, the fix that went in in is going to cause us data issues we don't want
Are you working on this? What's the timeline to resolution?
Based on above comment, looks like actually introduced a new regression. Am I right?
In , I reverted what's introduced in .
- is working on it
I approved this for 1.7 based on comment at We should revert , but are we ok with since it also introducing a new behavior. I will leave to decide.
replaces with a fix that addresses this issue systemically. While this bug did exist in 1.6.x, more things rely on GC in 1.7 than in 1.6, and without that fix, GC leaves more resources vulnerable to unintentional orphaning. Since the issue is actually with general GC behavior, should weigh in.


Question: "context" "crypto/sha256" "encoding/binary" <del> "encoding/json" </del> "errors" "fmt" "math"

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: cluster/aws/templates/configure-vm-aws.sh:  env-to-grains "runtime_config" cluster/aws/templates/configure-vm-aws.sh:  kubelet_api_servers: '${KUBELET_APISERVER}' cluster/centos/util.sh:  local node_ip=${node#*@} <ins> cluster/gce/config-default.sh:# reloads <os_distro>/helper.sh in the gap between when the master is created </ins> cluster/gce/configure-vm.sh:  advertise_address: '${EXTERNAL_IP}' cluster/gce/configure-vm.sh:  api_servers: '${KUBERNETES_MASTER_NAME}' cluster/gce/configure-vm.sh:  cloud_config: ${CLOUD_CONFIG}

We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.

at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far.

:smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:


Question: c.FuzzNoCustom(obj) obj.BindAddress = fmt.Sprintf("%d.%d.%d.%d", c.Intn(256), c.Intn(256), c.Intn(256), c.Intn(256)) obj.ClientConnection.ContentType = c.RandString() <ins> obj.DetectLocal.ClusterCIDRs = getRandomDualStackCIDR(c) </ins> obj.Linux.Conntrack.MaxPerCore = ptr.To(c.Int31()) obj.Linux.Conntrack.Min = ptr.To(c.Int31()) obj.Linux.Conntrack.TCPCloseWaitTimeout = &metav1.Duration{Duration: time.Duration(c.Int63()) * time.Hour}

pull-kubernetes-unit TestRoundTripFuzzing Only noticed the failure once in No response No response No response
/sig network may be caused by
This flakes in /cc
/assign so the problem is that sometimes the fuzzer generates a random string value that contains a comma, and when that gets converted back from v1alpha1, the comma is treated as punctuation rather than as part of the value. we should probably manually fuzz ClusterCIDRs to contain 0-2 actual CIDR values
/triage accepted
Saw here:


Question: } storage := pv.Spec.Capacity[api.ResourceStorage] w.Write(LEVEL_0, "Capacity:t%sn", storage.String()) <ins> printVolumeNodeAffinity(w, pv.Spec.NodeAffinity) </ins> w.Write(LEVEL_0, "Message:t%sn", pv.Status.Message) w.Write(LEVEL_0, "Source:n")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Now that it's a field, it needs a separate entry in 'kubectl describe'


Question: endpointsHandler.ValidateRequestCount(t, 0) } <ins> func TestCheckLeftoverEndpoints(t *testing.T) { ns := api.NamespaceDefault // Note that this requests *all* endpoints, therefore the NamespaceAll // below. testServer, _ := makeTestServer(t, api.NamespaceAll, serverResponse{http.StatusOK, &api.EndpointsList{ ListMeta: api.ListMeta{ ResourceVersion: "1", }, Items: []api.Endpoints{{ ObjectMeta: api.ObjectMeta{ Name:            "foo", Namespace:       ns, ResourceVersion: "1", }, Subsets: []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{IP: "6.7.8.9"}}, Ports:     []api.EndpointPort{{Port: 1000}}, }}, }}, }}) defer testServer.Close() client := client.NewOrDie(&client.Config{Host: testServer.URL, Version: testapi.Version()}) endpoints := NewEndpointController(client) endpoints.checkLeftoverEndpoints() if e, a := 1, endpoints.queue.Len(); e != a { t.Fatalf("Expected %v, got %v", e, a) } got, _ := endpoints.queue.Get() if e, a := ns+"/foo", got; e != a { t.Errorf("Expected %v, got %v", e, a) } } </ins> func TestSyncEndpointsProtocolTCP(t *testing.T) { ns := "other" testServer, endpointsHandler := makeTestServer(t, ns,

So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.


Question: "k8s.io/api/core/v1" "k8s.io/client-go/tools/record" "k8s.io/kubernetes/pkg/kubelet/cadvisor" <ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" </ins> "k8s.io/kubernetes/pkg/util/mount" )

https://k8s- https://k8s- cc: /kind bug /priority failing-test
Thanks a lot for reporting this! I think I have a fix PR for this issue. Or should we roll PR back?
I've kicked off testing of your PR. If it fixes the cross build, then that may be the easier path than reverting. cc: (sig-node leads)
I also fixed it in since it also touches the imports in that file. Would love to not rebase it yet again for this.
That call would be up to or they would have to determine which approach they prefer (fix on it's own in a small PR, or fix as part of a larger PR to prevent rebase).
Smaller is probably better; that way if the combined one needs to be reverted it doesn't re-break cross as a result. Will rebase if necessary (will depend on queue order.)


Question: import ( "flag" "fmt" <ins> "net" </ins> "net/http" <ins> "strconv" </ins> "time" "github.com/golang/glog"

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: KUBELET_PID=$! PROXY_LOG=/tmp/kube-proxy.log <del> "${GO_OUT}/kube-proxy"  </del> <ins> sudo "${GO_OUT}/kube-proxy"  </ins> -v=${LOG_LEVEL}  --master="http://${API_HOST}:${API_PORT}" >"${PROXY_LOG}" 2>&1 & PROXY_PID=$! SCHEDULER_LOG=/tmp/kube-scheduler.log <del> "${GO_OUT}/kube-scheduler"  </del> <ins> sudo "${GO_OUT}/kube-scheduler"  </ins> -v=${LOG_LEVEL}  --master="http://${API_HOST}:${API_PORT}" >"${SCHEDULER_LOG}" 2>&1 & SCHEDULER_PID=$!

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


Question: "Histogram": gstruct.Ignore(), })) } <ins> func haveKeys(keys ...string) types.GomegaMatcher { gomega.ExpectWithOffset(1, keys).ToNot(gomega.BeEmpty()) matcher := gomega.HaveKey(keys[0]) if len(keys) == 1 { return matcher } for _, key := range keys[1:] { matcher = gomega.And(matcher, gomega.HaveKey(key)) } return matcher } </ins>

Node e2e features is failing after PR for graduating swap to beta is merged. After is merged. Test needs to be adjusted for the extra field for swap usage. No response /sig node
/cc
/assign
/triage accepted /priority critical-urgent


Question: } func (f *fakePodKiller) killPodNow(pod *v1.Pod, status v1.PodStatus, gracePeriodOverride *int64) error { <ins> if f.errDuringPodKilling { f.killedPods = []*v1.Pod{} return fmt.Errorf("problem killing pod %v", pod) } </ins> f.killedPods = append(f.killedPods, pod) return nil }

What happened: Kops runs from a pod manifest stored on the node, having the annotation. We've experienced the case where the cluster autoscaler turned on a new node and, once the node has been to the cluster, application pods have been admitted before . Once it was the turn of , all system resources were utilized, so the pod has not been regularly admitted. However, since it's a critical pod, the executed the , which has tried to kill () a pod whose container wasn't started yet (it was downloading the Docker image), which has caused to timeout and return the error . The admission of failed with the following error: The pod execution has not been retried, so we've got a node without running and its networking broken. What you expected to happen: The should be eventually admitted. Anything else we need to know?: A minimal log extraction for comprehension: From the "Failed to admit pod" log I have been able to trace back the core code involved: () Logs: "Failed to admit pod [POD] - [ERROR]" () Error message: "preemption: pod [POD] failed to evict [ERROR]" () -() Error message: "timeout waiting to kill pod" Environment: Kubernetes version: Cloud provider: AWS OS: ubuntu 16.04.6 LTS (Xenial Xerus) Kernel: Install tools: kops Network plugin and version: kubenet
/sig node
I think the following change may alleviate the situation you encountered:
Thanks for your reply. To my understanding, when the is called it will return the minimum set of pods required to free up resources, so even if only 1 pod eviction fails it's likely that at the end of the the will not have enough resources anyway to admit the critical pod. In your opinion / experience: be an option introducing retries in the in case the eviction of 1 pod fails? yes, do you have any suggestion of a similar design adopted somewhere else in the ?
Looking at getPodsToPreempt, I am not sure it returns the minimum set of Pods. For , retrying may not help much since the pod to be killed may not respond to the subsequent kill command.
Sorry to reiterate on this, but I double checked the code and - if I'm not missing anything - returns the minimum set of pods.

Let's look at logic: pods by QOS class using that the total sum of evictable pods resources is enough to satify the requirements (resources to evict for the critical pod) the list of pods to evict, as the merge of three sublists: : the minimum set of pods to evict assuming we've already evicted all and all : the minimum set of pods to evict assuming we've already evicted all and the required : the minimum set of pods to evict to satify the requirements left Am I missing anything? A possible different approach to solve the issue is to refactor to return all input pods sorted by "should be evicted first". Then iterates over the full sorted list of potential pods to evict until the requirements are not satisfied (skipping to subtract pod resources from requirements if a pod fails on ) and breaking the loop as soon as the requirements are satisfied. What's your take on this? /cc who originally implemented this
w.r.t. approach outlined above, it makes sense.
Whether this evicts the "minimum set" depends on how you define the "minimum set". At the time this was implemented (k8s v1.5.0 IIRC), we didn't have PodPriority, so we used QoS to mirror the kubelet's eviction algorithm. It should probably prioritize low-priority pods for eviction today given the recent changes to the eviction algorithm to incorporate PodPriority. The current approach minimizes the eviction of guaranteed pods, and prioritizes evicting few larger pods over many smaller ones. Since eviction marks the evicted pod as failed, it should suffice to just admit the pod regardless of what is returned by the evictPod function. In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. It would eliminate this class of bugs entirely, which was the purpose of the feature in the first place.
See if the following aligns with your suggestion:
Something like that, yeah. I'm not sure the "Failed to admit pod" log warning is still required, but we can iron those things out during code review.
Doing like this it means we do always admit pods, while we should do it only for critical pods, correct?
Please see this The change is for critical pod(s).