Question: return matchingInstances, nil } <del> // getAllZones retrieves  a list of all the zones in which nodes are running </del> <ins> // getCandidateZonesForDynamicVolume retrieves  a list of all the zones in which nodes are running </ins> // It currently involves querying all instances <del> func (c *Cloud) getAllZones() (sets.String, error) { </del> <ins> func (c *Cloud) getCandidateZonesForDynamicVolume() (sets.String, error) { </ins> // We don't currently cache this; it is currently used only in volume // creation which is expected to be a comparatively rare occurrence.

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Install tools: default install What happened: AWS EBS backed PersitantVolumeClaim created a volume in an AZ where there was a master but no nodes. Making it unusable. What you expected to happen: Zones with master and no nodes shouldn't be used to create the EBS Volume How to reproduce it (as minimally and precisely as possible): AWS setup across 3 AZs in one region Master in three AZs Nodes in 2 AZs (so one AZ has a Master but no Nodes) Create a Storage class to use EBS without specifying AZCreate PVC against StorageClassPVC can create a volume in an availability zone with no nodes Anything else do we need to know: Mentioned on Slack channel #sig-aws with who asked me to raise this
, Mengqi Yu wrote:
Running Kubernetes master in a zone that has no running nodes is quite strange and I don't think this is a typical case we should support. AWS cloud provider lists all machines in the cluster and chooses a random zone that has at least one running instance. It seems it does not distinguish master/node machines. IMO, GCE has the same problem, it lists all zones in a region where the master runs and it does not distinguish masters and nodes either. We can either introduce a new AWS instance tag with master/node role that is running on the instance or we can move the zone decision into volume plugin itself, where we have a kubeclient and we can filter nodes relatively easily, however we will depend on an admission controller that must add a zone label to nodes. thoughts? As a third option, we could rely on kubernetes admin to fill all zones into StorageClass in this weird case. Right now we support only one zone in "zone" parameter, we could allow a list:
just to add to this, the reason why it's a deal is I actually don't have full control over where the nodes run, as they're in an autoscale group across all AZs. So you could have master in AZ a and nodes in b and c just because that's what the auto scale group picked.
I just ran into this, however, I only have nodes / masters in 3 zones, but the PV was created in a 4th zone.

So right now I can't even spin up a node in the last AZ since in my infrastructure, I'm not supporting it.
I think it may be more than just where there are or not nodes vs masters in an AZ. As an additional use case, consider pods that must run on a "dedicated" tenancy node in AWS, (which have a baseline cost per-zone) for compliance. Since masters are supposed to be an odd number (i.e. 3 zones) it would make sense to have 3 instead of 2. But for a pod that has to be scheduled on a "dedicated" tenancy node, there may be only 2 zones with nodes that are "dedicated" due to said cost. Even if there are nodes in all zones, a pod may need to be scheduled on a specific node or subset thereof. More information about node selection: More information about why this specific case would be necessary here: If PVs are used by pods, and pods must be able to access them from whatever nodes they are constrained/scheduled to, I think whatever solution is implemented needs to account for that.
do you have any other non-Kubernetes instances in us-east-1b in your AWS account? Did you try to tag your AWS instances that run Kubernetes with tag - all AWS instances in one Kubernetes cluster must share the same tag value, so you can run multiple Kubernetes clusters in one AWS account. This tag is not documented, see .
There is another non-k8s instance running in us-east-1b, but it's not related (created via breanstalk). I labels to all my nodes / controller as you sugessted and still it created the PV in . I may bail on Statefulset for now and manage my pv's by hand since I'm building out an operator and it wont be that painful, but would like to see if this could be addressed. I'm also willing to help out if you can help point into the right places this logic lives. I'll also try and dig in and see if I can find it myself.
It would be ideal to have the StorageClass accept an expression or wildcard zone under the aws provisioner. Ideally I shouldn't have to care where my pods are running and where the EBS are created.
We've a PR that adds "zones:" with list of zones instead of single "zone:". No wildcards though.

I just hit this as well. I think moving the "zone" decision closer to the provisioning decision would be best. Perhaps wait until a PVC is consumed by a pod before you create it in a zone. That way corner cases like this are not a problem.
Just want to check whether PR is good enough to solve the issue raised here?
, is right. Another use case is that if pod consumes 1 PVC, associated PVs might be created in different zones, therefore pod becomes unschedulable, as there is no node which belongs 1 AZ. Zone decision when provisioning PV must be made as late as possible, while currently it seems to be a very first thing to do.
I don't think it can be closed yet, reasons are: those magical tags are not documented anywhere, how people or going to know to use them? can't it continue to create volume in a zone where pod with a given PVC can never be scheduled (lets say it has node affinity configured to a node in a particular AZ)?
Would it make sense to say that the dynamic creation of a volume should happen only once a PVC has been attached to a pod which has been successfully scheduled to a node? Then the av_zone of the volume should match that of the selected node?
Yes I would agree with that statement. The PVC should be created when we know the node and AZ it is a apart of.
Just to add another use case to this - I just spun up a new cluster in AWS. Masters spanning , , and . After creating the ASG for nodes, I learned that AWS is out of instance capacity in for that particular instance type. Now I currently don't have nodes in (until capacity is restored and I can move an instance over). The reason this is different is because, in this case, I don't want to tell k8s to only use a subset of AZs. I want it to deploy across all AZs that are available. It just so happens that at this moment in time not all AZs are available. suggestion above would work, I believe. Schedule the node first, then provision a volume. That might cause other problems, though.
I just ran into this today - PVC created in eu-west-1c, but the only node with memory to to run the pod was in eu-west-1b.

This was a new pod and a new volume claim. Can we have this re-opened at least please?
We had to ultimately get rid of all persistence volume claims on AWS because of this. For really large clusters with lots of reserved capacity in all availability zones that woulnd't be such a big deal, but I'd image most companies are trying to optimize their cost and don't have that many node pre-provisioned just for the sake of it. Still looking for a solution which is easy to configure and AZ-failure friendly
Right now, there isn't really. You have a few options: EFS - it is regional, not zonal. You have to be OK working with NFS as the protocol, and potential performance implications. To be fair, NFS has gotten quote good, I know companies using it to store critical data. I am not an NFS expert, so don't ask my opinions here... Commercial solutions - Portworx and StorageOS both provide local storage, backed by local disk or EBS, but replicated at the app level (their app) between nodes, across zones, so you don't depend on zonal resources (like EBS). DIY - have fun. Ceph, some other solutions. For one use case with which I am involved, we are living with it, and using EFS where necessary. We don't love it, but the resources to implement something else aren't there now.
Would in a StorageClass go some way to fixing the issue? (At least for new claims anyway)? From Edit: I tried this, and the kube-controller-manager logs say: but the pod is FailedScheduling "2 node(s) didn't find available persistent volumes to bind.".
I solved this in the Elastic Operator by creating StorageClasses per zone and spreading work across that way. You need to make sure your app is zone tolerant. For example, the Elastic operator sets up sharding and replication across zones, given one AZ goes down, the other two still have enough of the data to continue functioning.
doesn't each pod then need access to storage from a different class (appropriate for the zone it is in)? How does that work?
Just ran into this as well, running work load specific zones in 2 regions for development to save cost over 3, had an EBS volume provisioned in the zone with no instance.

should address the above scenarios when you use set in the storageclass. The plumbing for went into EBS in 1.12 as a beta feature.
I have a similar issue: 3 masters + 3 nodes in 3 regions. I have a PV (created on regionA) for data backups created by a CronJob. When the cron's job gets scheduled on a node on a different region (regionA's CPU has been allocated almost fully and the master chooses another node in a different region) - it can't start because the PV is not accessible. Is there a way to avoid this issue?
I am also facing the same issue with 1.10 version of Kubernetes. Nodes are running in two regions, PV got created in the same region. still, I am getting
As mentioned above, this feature is now in beta in 1.12:
Are there any workaround for Amazon EKS to address the above issue? I am trying to make a multi-az JupyterHub deployment and changing zones causes the "no available volume zone" error...
Is this still in progress? I just ran in to issue as well
we moved it to GCP with 1.10 version and didn't face the issue
I'm also having this issue... So there's no solution to this?
im also having this issue on AWS EKS Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m (x841 over 1h) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. losing hope over here :(
Warning FailedScheduling 3m2s (x660 over 58m) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone.
Another similar case here, like 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. Like the use case by but the reason is not memory is pods allocation. The EBS was created in the AZ where the node is full (1c).. you can guess the rest..
Any update on this ? Since Node affinity is a beta feature causing this issue?

Edit: (Beware: Updating your EKS cluster can potentially screw things up & cause nodes and volumes in different zones)
I'm using EKS to deploy my services integrating it with rancher, I ran into the same issue as mentioned above. My setup is something like this: have few labels for the worked nodes to seperate the nodes per namespace/team as each team requires different types of instances. created the nodes for the pods in that node needs persistant volumes in a single availability zone as aws EBS attachment is per availability zone. I deployed my applications, the persistant volume got created but in different availability zone as I used the default storage class which creates EBS in random zones. The issue is fixed by creating another storage class with only single availbility zone.
I am using EKS , and I designed not only my pods to be Cattle, but also my nodes are Cattle as well : I have a Pod created by statefulset. This statefulset defines a PVC. The PVC triggers the creation of new EBS in the same AZ of the Node (i don't have pb here) Because Nodes are Cattle (destoryable any time), it happen that there is no Node exist in the AZ of EBS (PVC) =Pod of statefulset remains in "Pending" state =Pod events shows this error: " 1 node(s) had no available volume zone" Once a New node joins the cluster in the wanted AZ, the Pod of statefulset runs like a charm because the EBS can be attached at that node (same AZ). Thinking to use EFS ( )but looking for better solutions.
from my experience, EFS performance is not that great. better avoid it.
It depends how you use it. It's not great for lots of little files (don't check out a git repo on it, for instance ). It's been working great for us in production for almost a year so far, mainly on larger PDF files etc. But, like anything, definitely test it for your use case.
So I do have a question. If I just put the PVC into the specific availability. zone. Does that allow the other pods on different nodes to use it?

The nodes running on other availability zones couldn't access this PV until if you use EFS rather than EBS
I ran into a similar issue today investigating a error I have a PVC: and originally it bound fine, with a PV in us-east-1d Then, I ran an upgrade and none of the new nodes were created in us-east-1d And, now I have a disk in a zone without any nodes.
I found the issue. The issue is that when you are using AWS then the PVC is set to the Node. That means that if you try to put that pod or deployment on another node then it will not connect with the PVCs and being a pending state. It's an AWS thing. I resolve this by specifying taints and toleration to the nodes within that specific region.
thank you! can you post an example for more elaboration


Question: "os" "path" "path/filepath" <ins> "regexp" </ins> "strings" "time"

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" v1core "k8s.io/client-go/kubernetes/typed/core/v1" <ins> "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/framework" e2ekubelet "k8s.io/kubernetes/test/e2e/framework/kubelet"

After , /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { t.Errorf("unexpected error: %v", err) } <del> if !strings.Contains(out, "bar") || !strings.Contains(out, "foo") || !strings.Contains(out, "foo.bar.com") || !strings.Contains(out, "/foo") { </del> <ins> if !strings.Contains(out, "bar") || !strings.Contains(out, "foo") || !strings.Contains(out, "foo.bar.com") || !strings.Contains(out, "/foo") || !strings.Contains(out, "app1") || !strings.Contains(out, "app2") { </ins> t.Errorf("unexpected out: %s", out) } }

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: var serviceColumns = []string{"NAME", "LABELS", "SELECTOR", "IP", "PORT"} var minionColumns = []string{"NAME", "LABELS"} var statusColumns = []string{"STATUS"} <del> var eventColumns = []string{"TIME", "NAME", "KIND", "CONDITION", "REASON", "MESSAGE"} </del> <ins> var eventColumns = []string{"TIME", "NAME", "KIND", "SUBOBJECT", "CONDITION", "REASON", "MESSAGE"} </ins> // addDefaultHandlers adds print handlers for default Kubernetes types. func (h *HumanReadablePrinter) addDefaultHandlers() {

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004


Question: } }) <ins> It("should be able to update NodePorts with two same port numbers but different protocols", func() { serviceName := "nodeport-update-service" ns := f.Namespace.Name jig := framework.NewServiceTestJig(cs, serviceName) By("creating a TCP service " + serviceName + " with type=ClusterIP in namespace " + ns) tcpService := jig.CreateTCPServiceOrFail(ns, nil) defer func() { framework.Logf("Cleaning up the updating NodePorts test service") err := cs.Core().Services(ns).Delete(serviceName, nil) Expect(err).NotTo(HaveOccurred()) }() jig.SanityCheckService(tcpService, v1.ServiceTypeClusterIP) svcPort := int(tcpService.Spec.Ports[0].Port) framework.Logf("service port TCP: %d", svcPort) // Change the services to NodePort and add a UDP port. By("changing the TCP service to type=NodePort and add a UDP port") newService := jig.UpdateServiceOrFail(ns, tcpService.Name, func(s *v1.Service) { s.Spec.Type = v1.ServiceTypeNodePort s.Spec.Ports = []v1.ServicePort{ { Name:     "tcp-port", Port:     80, Protocol: v1.ProtocolTCP, }, { Name:     "udp-port", Port:     80, Protocol: v1.ProtocolUDP, }, } }) jig.SanityCheckService(newService, v1.ServiceTypeNodePort) if len(newService.Spec.Ports) != 2 { framework.Failf("new service should have two Ports") } for _, port := range newService.Spec.Ports { if port.NodePort == 0 { framework.Failf("new service failed to allocate NodePort for Port %s", port.Name) } framework.Logf("new service allocates NodePort %d for Port %s", port.NodePort, port.Name) } }) </ins> It("should be able to change the type from ExternalName to ClusterIP", func() { serviceName := "externalname-service" ns := f.Namespace.Name

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/apimachinery/pkg/util/validation" </del> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/kubernetes/pkg/apis/core/helper" v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> apivalidation "k8s.io/kubernetes/pkg/apis/core/validation" </ins> ) const isNegativeErrorMsg string = `must be greater than or equal to 0`

After got merged, we have new helper function here: It will be good for us to use this simpler helper function to replace previous function. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : After got merged, we have new helper function here: </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig usability
I would like to take this one /assign
I worked in on this as well, but kept your changed files untouched (as written in my PR description). same for you. I hope this is OK for you.
This should be reopened until and got merged (my PR excluded the changes from the other PRs).
/reopen per last comment
Reopened this issue. <detailsIn response to : This should be reopened until and got merged (my PR excluded the changes from the other PRs). per last comment Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
can you assign me
/assign
/assign
I'd love to work on this issue.

/assign
Hi, , I am a first commitor for kubernetes and I have committed a PR in , could you help me to review it, or give me some suggestions. Thanks a lot.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return volumes[0], nil } <ins> // waitForAttachmentStatus polls until the attachment status is the expected value // TODO(justinsb): return (bool, error) </ins> func (self *awsDisk) waitForAttachmentStatus(status string) error { // TODO: There may be a faster way to get this when we're attaching locally attempt := 0

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: Description: Make sure a container's subpath can not be set using an expansion of environment variables when absolute path is supplied. */ framework.ConformanceIt("should fail substituting values in a volume subpath with absolute path [sig-storage][Slow]", func() { <ins> absolutePath := "/tmp" if framework.NodeOSDistroIs("windows") { // Windows does not typically have a C:tmp folder. absolutePath = "C:Users" } </ins> envVars := []v1.EnvVar{ { Name:  "POD_NAME", <del> Value: "/tmp", </del> <ins> Value: absolutePath, </ins> }, } mounts := []v1.VolumeMount{

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: N/A Which test(s) are failing: should fail substituting values in a volume subpath with absolute path Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Some of the tests are negative test cases which are supposed to ensure that those invalid usecases are handled properly. However, some of the tests are false positives, they can pass due to various reasons. One such example is: should fail substituting values in a volume subpath with absolute path. This test can pass if: the Pod cannot start due to various reasons (e.g.: the container image cannot be pulled or does not exist). the Pod ran to completion, even though the container was not supposed to start in the first place. Anything else we need to know: /sig testing Related:
/sig storage SIG storage owns the test case, though I see that you are changing core framework methods to fix this, so leaving SIG testing. Looks like also possibly sig-node, skimming the linked PRs Based on the PRs this also appears to be related to windows? (unsure, no mention here)
I've Related: to the issue description. After fixing this test, I've noticed that it now fails on Windows, which means the issue was masked by the test's false positive result. Basically, it means that you can have volume subpaths with absolute paths substituted on variable expansion and have Windows containers use them and be running, which according to the test, it shouldn't be a supported usecase. The fix should probably be also backported to previous releases.


Question: klog.V(4).InfoS("Validating the CRI v1 API runtime version") r.runtimeClient = runtimeapi.NewRuntimeServiceClient(conn) <del> if _, err := r.runtimeClient.Version(ctx, &runtimeapi.VersionRequest{}); err == nil { klog.V(2).InfoS("Validated CRI v1 runtime API") } else if status.Code(err) == codes.Unimplemented { return fmt.Errorf("CRI v1 runtime API is not implemented for endpoint %q: %w", endpoint, err) </del> <ins> if _, err := r.runtimeClient.Version(ctx, &runtimeapi.VersionRequest{}); err != nil { return fmt.Errorf("validate CRI v1 runtime API for endpoint %q: %w", endpoint, err) </ins> } <ins> klog.V(2).InfoS("Validated CRI v1 runtime API") </ins> return nil }

It "connects" to dockershim There is no Run crictl v1.26.0, without setting up (or using -r) Worked in crictl v1.25.0, that still had CRI v1alpha2 This patch fixed the issue, to make the fallbacks work: <detailsv1.26.0 </details<detailsLima </details<detailsPRETTY_NAME="Ubuntu 22.04.1 LTS" </details<details</details<detailsv1.6.12 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I agree that we can check for unavailable connection because it will only be done on kubelet startup.


Question: <ins> filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [ ":package-srcs", "//vendor/bitbucket.org/bertimus9/systemstat:all-srcs", "//vendor/bitbucket.org/ww/goautoneg:all-srcs", "//vendor/cloud.google.com/go/compute/metadata:all-srcs", "//vendor/cloud.google.com/go/internal:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/arm/compute:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/arm/containerregistry:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/arm/disk:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/arm/network:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/arm/storage:all-srcs", "//vendor/github.com/Azure/azure-sdk-for-go/storage:all-srcs", "//vendor/github.com/Azure/go-ansiterm:all-srcs", "//vendor/github.com/Azure/go-autorest/autorest:all-srcs", "//vendor/github.com/MakeNowJust/heredoc:all-srcs", "//vendor/github.com/Microsoft/go-winio:all-srcs", "//vendor/github.com/NYTimes/gziphandler:all-srcs", "//vendor/github.com/PuerkitoBio/purell:all-srcs", "//vendor/github.com/PuerkitoBio/urlesc:all-srcs", "//vendor/github.com/Sirupsen/logrus:all-srcs", "//vendor/github.com/abbot/go-http-auth:all-srcs", "//vendor/github.com/appc/spec/schema:all-srcs", "//vendor/github.com/armon/circbuf:all-srcs", "//vendor/github.com/asaskevich/govalidator:all-srcs", "//vendor/github.com/aws/aws-sdk-go/aws:all-srcs", "//vendor/github.com/aws/aws-sdk-go/private/protocol:all-srcs", "//vendor/github.com/aws/aws-sdk-go/private/waiter:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/autoscaling:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/ec2:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/ecr:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/elb:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/route53:all-srcs", "//vendor/github.com/aws/aws-sdk-go/service/sts:all-srcs", "//vendor/github.com/beorn7/perks/quantile:all-srcs", "//vendor/github.com/blang/semver:all-srcs", "//vendor/github.com/boltdb/bolt:all-srcs", "//vendor/github.com/chai2010/gettext-go/gettext:all-srcs", "//vendor/github.com/cloudflare/cfssl/auth:all-srcs", "//vendor/github.com/cloudflare/cfssl/certdb:all-srcs", "//vendor/github.com/cloudflare/cfssl/config:all-srcs", "//vendor/github.com/cloudflare/cfssl/crypto/pkcs7:all-srcs", "//vendor/github.com/cloudflare/cfssl/csr:all-srcs", "//vendor/github.com/cloudflare/cfssl/errors:all-srcs", "//vendor/github.com/cloudflare/cfssl/helpers:all-srcs", "//vendor/github.com/cloudflare/cfssl/info:all-srcs", "//vendor/github.com/cloudflare/cfssl/log:all-srcs", "//vendor/github.com/cloudflare/cfssl/ocsp/config:all-srcs", "//vendor/github.com/cloudflare/cfssl/signer:all-srcs", "//vendor/github.com/clusterhq/flocker-go:all-srcs", "//vendor/github.com/codedellemc/goscaleio:all-srcs", "//vendor/github.com/codegangsta/negroni:all-srcs", "//vendor/github.com/containernetworking/cni/libcni:all-srcs", "//vendor/github.com/containernetworking/cni/pkg/invoke:all-srcs", "//vendor/github.com/containernetworking/cni/pkg/types:all-srcs", "//vendor/github.com/containernetworking/cni/pkg/version:all-srcs", "//vendor/github.com/coreos/etcd/alarm:all-srcs", "//vendor/github.com/coreos/etcd/auth:all-srcs", "//vendor/github.com/coreos/etcd/client:all-srcs", "//vendor/github.com/coreos/etcd/clientv3:all-srcs", "//vendor/github.com/coreos/etcd/compactor:all-srcs", "//vendor/github.com/coreos/etcd/discovery:all-srcs", "//vendor/github.com/coreos/etcd/error:all-srcs", "//vendor/github.com/coreos/etcd/etcdserver:all-srcs", "//vendor/github.com/coreos/etcd/integration:all-srcs", "//vendor/github.com/coreos/etcd/lease:all-srcs", "//vendor/github.com/coreos/etcd/mvcc:all-srcs", "//vendor/github.com/coreos/etcd/pkg/adt:all-srcs", "//vendor/github.com/coreos/etcd/pkg/contention:all-srcs", "//vendor/github.com/coreos/etcd/pkg/cpuutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/crc:all-srcs", "//vendor/github.com/coreos/etcd/pkg/fileutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/httputil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/idutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/ioutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/logutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/monotime:all-srcs", "//vendor/github.com/coreos/etcd/pkg/netutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/pathutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/pbutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/runtime:all-srcs", "//vendor/github.com/coreos/etcd/pkg/schedule:all-srcs", "//vendor/github.com/coreos/etcd/pkg/testutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/tlsutil:all-srcs", "//vendor/github.com/coreos/etcd/pkg/transport:all-srcs", "//vendor/github.com/coreos/etcd/pkg/types:all-srcs", "//vendor/github.com/coreos/etcd/pkg/wait:all-srcs", "//vendor/github.com/coreos/etcd/proxy/grpcproxy:all-srcs", "//vendor/github.com/coreos/etcd/raft:all-srcs", "//vendor/github.com/coreos/etcd/rafthttp:all-srcs", "//vendor/github.com/coreos/etcd/snap:all-srcs", "//vendor/github.com/coreos/etcd/store:all-srcs", "//vendor/github.com/coreos/etcd/version:all-srcs", "//vendor/github.com/coreos/etcd/wal:all-srcs", "//vendor/github.com/coreos/go-oidc/http:all-srcs", "//vendor/github.com/coreos/go-oidc/jose:all-srcs", "//vendor/github.com/coreos/go-oidc/key:all-srcs", "//vendor/github.com/coreos/go-oidc/oauth2:all-srcs", "//vendor/github.com/coreos/go-oidc/oidc:all-srcs", "//vendor/github.com/coreos/go-semver/semver:all-srcs", "//vendor/github.com/coreos/go-systemd/daemon:all-srcs", "//vendor/github.com/coreos/go-systemd/dbus:all-srcs", "//vendor/github.com/coreos/go-systemd/journal:all-srcs", "//vendor/github.com/coreos/go-systemd/unit:all-srcs", "//vendor/github.com/coreos/go-systemd/util:all-srcs", "//vendor/github.com/coreos/pkg/capnslog:all-srcs", "//vendor/github.com/coreos/pkg/dlopen:all-srcs", "//vendor/github.com/coreos/pkg/health:all-srcs", "//vendor/github.com/coreos/pkg/httputil:all-srcs", "//vendor/github.com/coreos/pkg/timeutil:all-srcs", "//vendor/github.com/coreos/rkt/api/v1alpha:all-srcs", "//vendor/github.com/cpuguy83/go-md2man/md2man:all-srcs", "//vendor/github.com/davecgh/go-spew/spew:all-srcs", "//vendor/github.com/daviddengcn/go-colortext:all-srcs", "//vendor/github.com/dgrijalva/jwt-go:all-srcs", "//vendor/github.com/docker/distribution/digest:all-srcs", "//vendor/github.com/docker/distribution/reference:all-srcs", "//vendor/github.com/docker/docker/api/types:all-srcs", "//vendor/github.com/docker/docker/client:all-srcs", "//vendor/github.com/docker/docker/pkg/jsonlog:all-srcs", "//vendor/github.com/docker/docker/pkg/jsonmessage:all-srcs", "//vendor/github.com/docker/docker/pkg/longpath:all-srcs", "//vendor/github.com/docker/docker/pkg/mount:all-srcs", "//vendor/github.com/docker/docker/pkg/stdcopy:all-srcs", "//vendor/github.com/docker/docker/pkg/symlink:all-srcs", "//vendor/github.com/docker/docker/pkg/system:all-srcs", "//vendor/github.com/docker/docker/pkg/term:all-srcs", "//vendor/github.com/docker/docker/pkg/tlsconfig:all-srcs", "//vendor/github.com/docker/engine-api/client:all-srcs", "//vendor/github.com/docker/engine-api/types:all-srcs", "//vendor/github.com/docker/go-connections/nat:all-srcs", "//vendor/github.com/docker/go-connections/sockets:all-srcs", "//vendor/github.com/docker/go-connections/tlsconfig:all-srcs", "//vendor/github.com/docker/go-units:all-srcs", "//vendor/github.com/docker/spdystream:all-srcs", "//vendor/github.com/elazarl/go-bindata-assetfs:all-srcs", "//vendor/github.com/elazarl/goproxy:all-srcs", "//vendor/github.com/emicklei/go-restful:all-srcs", "//vendor/github.com/emicklei/go-restful-swagger12:all-srcs", "//vendor/github.com/evanphx/json-patch:all-srcs", "//vendor/github.com/exponent-io/jsonpath:all-srcs", "//vendor/github.com/fatih/camelcase:all-srcs", "//vendor/github.com/fsnotify/fsnotify:all-srcs", "//vendor/github.com/garyburd/redigo/internal:all-srcs", "//vendor/github.com/garyburd/redigo/redis:all-srcs", "//vendor/github.com/ghodss/yaml:all-srcs", "//vendor/github.com/go-ini/ini:all-srcs", "//vendor/github.com/go-openapi/analysis:all-srcs", "//vendor/github.com/go-openapi/errors:all-srcs", "//vendor/github.com/go-openapi/jsonpointer:all-srcs", "//vendor/github.com/go-openapi/jsonreference:all-srcs", "//vendor/github.com/go-openapi/loads:all-srcs", "//vendor/github.com/go-openapi/runtime:all-srcs", "//vendor/github.com/go-openapi/spec:all-srcs", "//vendor/github.com/go-openapi/strfmt:all-srcs", "//vendor/github.com/go-openapi/swag:all-srcs", "//vendor/github.com/go-openapi/validate:all-srcs", "//vendor/github.com/godbus/dbus:all-srcs", "//vendor/github.com/gogo/protobuf/gogoproto:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/compare:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/defaultcheck:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/description:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/embedcheck:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/enumstringer:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/equal:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/face:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/gostring:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/marshalto:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/oneofcheck:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/populate:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/size:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/stringer:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/testgen:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/union:all-srcs", "//vendor/github.com/gogo/protobuf/plugin/unmarshal:all-srcs", "//vendor/github.com/gogo/protobuf/proto:all-srcs", "//vendor/github.com/gogo/protobuf/protoc-gen-gogo/descriptor:all-srcs", "//vendor/github.com/gogo/protobuf/protoc-gen-gogo/generator:all-srcs", "//vendor/github.com/gogo/protobuf/protoc-gen-gogo/grpc:all-srcs", "//vendor/github.com/gogo/protobuf/protoc-gen-gogo/plugin:all-srcs", "//vendor/github.com/gogo/protobuf/sortkeys:all-srcs", "//vendor/github.com/gogo/protobuf/vanity:all-srcs", "//vendor/github.com/golang/glog:all-srcs", "//vendor/github.com/golang/groupcache/lru:all-srcs", "//vendor/github.com/golang/mock/gomock:all-srcs", "//vendor/github.com/golang/protobuf/jsonpb:all-srcs", "//vendor/github.com/golang/protobuf/proto:all-srcs", "//vendor/github.com/golang/protobuf/ptypes:all-srcs", "//vendor/github.com/google/btree:all-srcs", "//vendor/github.com/google/cadvisor/api:all-srcs", "//vendor/github.com/google/cadvisor/cache/memory:all-srcs", "//vendor/github.com/google/cadvisor/client/v2:all-srcs", "//vendor/github.com/google/cadvisor/collector:all-srcs", "//vendor/github.com/google/cadvisor/container:all-srcs", "//vendor/github.com/google/cadvisor/devicemapper:all-srcs", "//vendor/github.com/google/cadvisor/events:all-srcs", "//vendor/github.com/google/cadvisor/fs:all-srcs", "//vendor/github.com/google/cadvisor/healthz:all-srcs", "//vendor/github.com/google/cadvisor/http:all-srcs", "//vendor/github.com/google/cadvisor/info/v1:all-srcs", "//vendor/github.com/google/cadvisor/info/v2:all-srcs", "//vendor/github.com/google/cadvisor/machine:all-srcs", "//vendor/github.com/google/cadvisor/manager:all-srcs", "//vendor/github.com/google/cadvisor/metrics:all-srcs", "//vendor/github.com/google/cadvisor/pages:all-srcs", "//vendor/github.com/google/cadvisor/storage:all-srcs", "//vendor/github.com/google/cadvisor/summary:all-srcs", "//vendor/github.com/google/cadvisor/utils:all-srcs", "//vendor/github.com/google/cadvisor/validate:all-srcs", "//vendor/github.com/google/cadvisor/version:all-srcs", "//vendor/github.com/google/cadvisor/zfs:all-srcs", "//vendor/github.com/google/certificate-transparency/go:all-srcs", "//vendor/github.com/google/gofuzz:all-srcs", "//vendor/github.com/googleapis/gnostic/OpenAPIv2:all-srcs", "//vendor/github.com/googleapis/gnostic/compiler:all-srcs", "//vendor/github.com/googleapis/gnostic/extensions:all-srcs", "//vendor/github.com/gophercloud/gophercloud:all-srcs", "//vendor/github.com/gorilla/context:all-srcs", "//vendor/github.com/gorilla/mux:all-srcs", "//vendor/github.com/gorilla/websocket:all-srcs", "//vendor/github.com/grpc-ecosystem/go-grpc-prometheus:all-srcs", "//vendor/github.com/grpc-ecosystem/grpc-gateway/runtime:all-srcs", "//vendor/github.com/grpc-ecosystem/grpc-gateway/utilities:all-srcs", "//vendor/github.com/hashicorp/golang-lru:all-srcs", "//vendor/github.com/hashicorp/hcl:all-srcs", "//vendor/github.com/hawkular/hawkular-client-go/metrics:all-srcs", "//vendor/github.com/heketi/heketi/client/api/go-client:all-srcs", "//vendor/github.com/heketi/heketi/pkg/glusterfs/api:all-srcs", "//vendor/github.com/heketi/heketi/pkg/utils:all-srcs", "//vendor/github.com/howeyc/gopass:all-srcs", "//vendor/github.com/imdario/mergo:all-srcs", "//vendor/github.com/inconshreveable/mousetrap:all-srcs", "//vendor/github.com/influxdata/influxdb/client:all-srcs", "//vendor/github.com/influxdata/influxdb/models:all-srcs", "//vendor/github.com/influxdata/influxdb/pkg/escape:all-srcs", "//vendor/github.com/jmespath/go-jmespath:all-srcs", "//vendor/github.com/jonboulle/clockwork:all-srcs", "//vendor/github.com/jteeuwen/go-bindata:all-srcs", "//vendor/github.com/juju/ratelimit:all-srcs", "//vendor/github.com/kardianos/osext:all-srcs", "//vendor/github.com/karlseguin/ccache:all-srcs", "//vendor/github.com/kr/fs:all-srcs", "//vendor/github.com/kr/pty:all-srcs", "//vendor/github.com/libopenstorage/openstorage/api:all-srcs", "//vendor/github.com/libopenstorage/openstorage/pkg/units:all-srcs", "//vendor/github.com/libopenstorage/openstorage/volume:all-srcs", "//vendor/github.com/lpabon/godbc:all-srcs", "//vendor/github.com/magiconair/properties:all-srcs", "//vendor/github.com/mailru/easyjson/buffer:all-srcs", "//vendor/github.com/mailru/easyjson/jlexer:all-srcs", "//vendor/github.com/mailru/easyjson/jwriter:all-srcs", "//vendor/github.com/matttproud/golang_protobuf_extensions/pbutil:all-srcs", "//vendor/github.com/miekg/coredns/middleware/etcd/msg:all-srcs", "//vendor/github.com/miekg/dns:all-srcs", "//vendor/github.com/mistifyio/go-zfs:all-srcs", "//vendor/github.com/mitchellh/go-wordwrap:all-srcs", "//vendor/github.com/mitchellh/mapstructure:all-srcs", "//vendor/github.com/mreiferson/go-httpclient:all-srcs", "//vendor/github.com/mvdan/xurls:all-srcs", "//vendor/github.com/mxk/go-flowrate/flowrate:all-srcs", "//vendor/github.com/onsi/ginkgo:all-srcs", "//vendor/github.com/onsi/gomega:all-srcs", "//vendor/github.com/opencontainers/runc/libcontainer:all-srcs", "//vendor/github.com/pborman/uuid:all-srcs", "//vendor/github.com/pelletier/go-buffruneio:all-srcs", "//vendor/github.com/pelletier/go-toml:all-srcs", "//vendor/github.com/pkg/errors:all-srcs", "//vendor/github.com/pkg/sftp:all-srcs", "//vendor/github.com/pmezard/go-difflib/difflib:all-srcs", "//vendor/github.com/prometheus/client_golang/prometheus:all-srcs", "//vendor/github.com/prometheus/client_model/go:all-srcs", "//vendor/github.com/prometheus/common/expfmt:all-srcs", "//vendor/github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg:all-srcs", "//vendor/github.com/prometheus/common/model:all-srcs", "//vendor/github.com/prometheus/procfs:all-srcs", "//vendor/github.com/quobyte/api:all-srcs", "//vendor/github.com/rackspace/gophercloud:all-srcs", "//vendor/github.com/rancher/go-rancher/client:all-srcs", "//vendor/github.com/renstrom/dedent:all-srcs", "//vendor/github.com/robfig/cron:all-srcs", "//vendor/github.com/rubiojr/go-vhd/vhd:all-srcs", "//vendor/github.com/russross/blackfriday:all-srcs", "//vendor/github.com/satori/uuid:all-srcs", "//vendor/github.com/seccomp/libseccomp-golang:all-srcs", "//vendor/github.com/shurcooL/sanitized_anchor_name:all-srcs", "//vendor/github.com/spf13/afero:all-srcs", "//vendor/github.com/spf13/cast:all-srcs", "//vendor/github.com/spf13/cobra:all-srcs", "//vendor/github.com/spf13/jwalterweatherman:all-srcs", "//vendor/github.com/spf13/pflag:all-srcs", "//vendor/github.com/spf13/viper:all-srcs", "//vendor/github.com/square/go-jose:all-srcs", "//vendor/github.com/storageos/go-api:all-srcs", "//vendor/github.com/stretchr/objx:all-srcs", "//vendor/github.com/stretchr/testify/assert:all-srcs", "//vendor/github.com/stretchr/testify/mock:all-srcs", "//vendor/github.com/stretchr/testify/require:all-srcs", "//vendor/github.com/syndtr/gocapability/capability:all-srcs", "//vendor/github.com/ugorji/go/codec:all-srcs", "//vendor/github.com/vishvananda/netlink:all-srcs", "//vendor/github.com/vmware/govmomi:all-srcs", "//vendor/github.com/vmware/photon-controller-go-sdk/SSPI:all-srcs", "//vendor/github.com/vmware/photon-controller-go-sdk/photon:all-srcs", "//vendor/github.com/xanzy/go-cloudstack/cloudstack:all-srcs", "//vendor/github.com/xiang90/probing:all-srcs", "//vendor/github.com/xyproto/simpleredis:all-srcs", "//vendor/go.pedge.io/pb/go/google/protobuf:all-srcs", "//vendor/go4.org/errorutil:all-srcs", "//vendor/golang.org/x/crypto/bcrypt:all-srcs", "//vendor/golang.org/x/crypto/blowfish:all-srcs", "//vendor/golang.org/x/crypto/curve25519:all-srcs", "//vendor/golang.org/x/crypto/ed25519:all-srcs", "//vendor/golang.org/x/crypto/nacl/secretbox:all-srcs", "//vendor/golang.org/x/crypto/pkcs12:all-srcs", "//vendor/golang.org/x/crypto/poly1305:all-srcs", "//vendor/golang.org/x/crypto/salsa20/salsa:all-srcs", "//vendor/golang.org/x/crypto/ssh:all-srcs", "//vendor/golang.org/x/exp/inotify:all-srcs", "//vendor/golang.org/x/net/context:all-srcs", "//vendor/golang.org/x/net/html:all-srcs", "//vendor/golang.org/x/net/http2:all-srcs", "//vendor/golang.org/x/net/idna:all-srcs", "//vendor/golang.org/x/net/internal/timeseries:all-srcs", "//vendor/golang.org/x/net/lex/httplex:all-srcs", "//vendor/golang.org/x/net/proxy:all-srcs", "//vendor/golang.org/x/net/trace:all-srcs", "//vendor/golang.org/x/net/websocket:all-srcs", "//vendor/golang.org/x/oauth2:all-srcs", "//vendor/golang.org/x/sys/unix:all-srcs", "//vendor/golang.org/x/sys/windows:all-srcs", "//vendor/golang.org/x/text/cases:all-srcs", "//vendor/golang.org/x/text/encoding:all-srcs", "//vendor/golang.org/x/text/internal:all-srcs", "//vendor/golang.org/x/text/language:all-srcs", "//vendor/golang.org/x/text/runes:all-srcs", "//vendor/golang.org/x/text/secure/bidirule:all-srcs", "//vendor/golang.org/x/text/secure/precis:all-srcs", "//vendor/golang.org/x/text/transform:all-srcs", "//vendor/golang.org/x/text/unicode/bidi:all-srcs", "//vendor/golang.org/x/text/unicode/norm:all-srcs", "//vendor/golang.org/x/text/width:all-srcs", "//vendor/golang.org/x/time/rate:all-srcs", "//vendor/golang.org/x/tools/container/intsets:all-srcs", "//vendor/google.golang.org/api/cloudkms/v1:all-srcs", "//vendor/google.golang.org/api/cloudmonitoring/v2beta2:all-srcs", "//vendor/google.golang.org/api/compute/v0.alpha:all-srcs", "//vendor/google.golang.org/api/compute/v0.beta:all-srcs", "//vendor/google.golang.org/api/compute/v1:all-srcs", "//vendor/google.golang.org/api/container/v1:all-srcs", "//vendor/google.golang.org/api/dns/v1:all-srcs", "//vendor/google.golang.org/api/gensupport:all-srcs", "//vendor/google.golang.org/api/googleapi:all-srcs", "//vendor/google.golang.org/api/logging/v2beta1:all-srcs", "//vendor/google.golang.org/api/monitoring/v3:all-srcs", "//vendor/google.golang.org/api/pubsub/v1:all-srcs", "//vendor/google.golang.org/genproto/googleapis/rpc/status:all-srcs", "//vendor/google.golang.org/grpc:all-srcs", "//vendor/gopkg.in/gcfg.v1:all-srcs", "//vendor/gopkg.in/inf.v0:all-srcs", "//vendor/gopkg.in/natefinch/lumberjack.v2:all-srcs", "//vendor/gopkg.in/warnings.v0:all-srcs", "//vendor/gopkg.in/yaml.v2:all-srcs", "//vendor/k8s.io/gengo/args:all-srcs", "//vendor/k8s.io/gengo/examples/deepcopy-gen/generators:all-srcs", "//vendor/k8s.io/gengo/examples/defaulter-gen/generators:all-srcs", "//vendor/k8s.io/gengo/examples/import-boss/generators:all-srcs", "//vendor/k8s.io/gengo/examples/set-gen/generators:all-srcs", "//vendor/k8s.io/gengo/examples/set-gen/sets:all-srcs", "//vendor/k8s.io/gengo/generator:all-srcs", "//vendor/k8s.io/gengo/namer:all-srcs", "//vendor/k8s.io/gengo/parser:all-srcs", "//vendor/k8s.io/gengo/types:all-srcs", "//vendor/k8s.io/heapster/metrics/api/v1/types:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/aggregator:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/builder:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/common:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/generators:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/handler:all-srcs", "//vendor/k8s.io/kube-openapi/pkg/util:all-srcs", "//vendor/k8s.io/utils/exec:all-srcs", "//vendor/vbom.ml/util/sortorder:all-srcs", ], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>

At this point I don't know if it is a valid file or not?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I reproduce it on my env. After removing , will not create and update ; Manually creating an empty file , will successfully rewrite and update ; /cc Seems file will not be created automatically if missing.
This is working as intended, or at least as designed. We only create new files if there are go sources in that directory, and doesn't have any go sources, so won't be auto-created. We have a solely for the and rules, but those are only managed in existing files. Without , we'd have a lot of lines in , which would be painful. I learned last week that the say to remove all of , which mucks with this process a little bit. I a step in the instructions there to recreate the file, but maybe should just be doing that? I'm not super excited about adding special cases there, but is kinda weird, so maybe it's unavoidable.
the more automatic the better
Specifically, this would cause many merge conflicts.
I'm a little confused, since this has been the way things have worked since merged in April, but 4+ people asking me about this in the last week have convinced me that this needs to be more automatic. Sent .


Question: t.Errorf(err.Error()) return } <del> actualStr := sortLines(string(data)) expectedStr := sortLines(expected) </del> <ins> actualStr := string(data) expectedStr := expected </ins> if actualStr != expectedStr { t.Errorf("Found `%s`, expected `%s`", actualStr, expectedStr) }

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "flag" "fmt" "io" "log" "os" "os/exec" "os/signal" "strings" "syscall" ) var ( logFilePath    = flag.String("log-file", "", "If non-empty, save stdout to this file") alsoToStdOut   = flag.Bool("also-stdout", false, "useful with log-file, log to standard output as well as the log file") redirectStderr = flag.Bool("redirect-stderr", true, "treat stderr same as stdout") ) func main() { flag.Parse() if err := configureAndRun(); err != nil { log.Fatal(err) } } func configureAndRun() error { var ( outputStream io.Writer = os.Stdout errStream    io.Writer = os.Stderr ) args := flag.Args() if len(args) == 0 { return fmt.Errorf("not enough arguments to run") } if logFilePath != nil && *logFilePath != "" { logFile, err := os.OpenFile(*logFilePath, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) if err != nil { return fmt.Errorf("failed to create log file %v: %w", *logFilePath, err) } if *alsoToStdOut { outputStream = io.MultiWriter(os.Stdout, logFile) } else { outputStream = logFile } } if *redirectStderr { errStream = outputStream } exe := args[0] var exeArgs []string if len(args) > 1 { exeArgs = args[1:] } cmd := exec.Command(exe, exeArgs...) cmd.Stdout = outputStream cmd.Stderr = errStream log.Printf("Running command:n%v", cmdInfo(cmd)) err := cmd.Start() if err != nil { return fmt.Errorf("starting command: %w", err) } // Handle signals and shutdown process gracefully. go setupSigHandler(cmd.Process) if err := cmd.Wait(); err != nil { return fmt.Errorf("running command: %w", err) } return nil } // cmdInfo generates a useful look at what the command is for printing/debug. func cmdInfo(cmd *exec.Cmd) string { return fmt.Sprintf( `Command env: (log-file=%v, also-stdout=%v, redirect-stderr=%v) Run from directory: %v Executable path: %v Args (comma-delimited): %v`, *logFilePath, *alsoToStdOut, *redirectStderr, cmd.Dir, cmd.Path, strings.Join(cmd.Args, ","), ) } // setupSigHandler will forward any termination signals to the process func setupSigHandler(process *os.Process) { // terminationSignals are signals that cause the program to exit in the // supported platforms (linux, darwin, windows). terminationSignals := []os.Signal{syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT} c := make(chan os.Signal, 1) signal.Notify(c, terminationSignals...) // Block until a signal is received. log.Println("Now listening for interrupts") s := <-c log.Printf("Got signal: %v. Sending down to process (PID: %v)", s, process.Pid) if err := process.Signal(s); err != nil { log.Fatalf("Failed to signal process: %v", err) } log.Printf("Signalled process %v successfully.", process.Pid) } </ins>

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for windows [ ] [ ] /sig windows /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I've started looking into this. Part of the problem is that go-runner is not part of kubernetes-node-windows- I suspect that is where the scripts get and kube- from.


Question: DESC="The etcd key-value share configuration service" NAME=etcd DAEMON=/usr/local/bin/$NAME <del> DAEMON_ARGS="-peer-addr $HOSTNAME:7001 -name $HOSTNAME" </del> <ins> # DAEMON_ARGS="-peer-addr $HOSTNAME:7001 -name $HOSTNAME" host_ip=$(hostname -i) DAEMON_ARGS="-addr ${host_ip}:4001 -bind-addr ${host_ip}:4001 -data-dir /var/etcd -initial-advertise-peer-urls http://${HOSTNAME}:2380 -name ${HOSTNAME} -initial-cluster ${HOSTNAME}=http://${HOSTNAME}:2380" </ins> DAEMON_LOG_FILE=/var/log/$NAME.log PIDFILE=/var/run/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME

writes: As we start looking into etcd and event memory pressure [], we should look into switching over to etcd 2.0 which was released today.
cc/
+1 On Jan 28, 2015 5:29 PM, "Dawn Chen" wrote:
My long-running cluster tests resulted in a hard-crash on etcd: We should move to 2.0 ASAP in the hopes of (a) fixing the issue and if not (b) prioritizing the fix since we'll be complaining about head. Abhi, is this something you can pick up and drive to land in O(days) please?
Calling this P0 as it blocks further long-running cluster stability testing.
I will do this today. , Alex Mohr wrote:


Question: framework.ExpectEqual(configMapFromUpdate.Data, configMap.Data) }) <del> ginkgo.It("should run through a ConfigMap lifecycle", func() { </del> <ins> /* Release : v1.19 Testname: ConfigMap lifecycle Description: Attempt to create a ConfigMap. Patch the created ConfigMap. Fetching the ConfigMap MUST reflect changes. By fetching all the ConfigMaps via a Label selector it MUST find the ConfigMap by it's static label and updated value. The ConfigMap must be deleted by Collection. */ framework.ConformanceIt("should run through a ConfigMap lifecycle", func() { </ins> testNamespaceName := f.Namespace.Name testConfigMapName := "test-configmap" + string(uuid.NewUUID())

[x] Original approval issue: [x] Two weeks soak end date: 8 May 2020 [x] Two weeks soak end date: 23 June 2020 This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. a ConfigMap with a static label the ConfigMap with a new Label and updated data the ConfigMap to ensure it's patched all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points
/sig testing /sig architecture /area conformance
/assign Issue created to replace
GitHub didn't allow me to assign the following users: raymonddeng99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen On hold & soaking
Reopened this issue. <detailsIn response to : /reopen On hold & soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } if len(strings.Split(stringValue, "/")) == 1 { if !standardFinalizers.Has(stringValue) { <del> allWarnings = append(allWarnings, fmt.Sprintf("%s: %q: prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers", fldPath.String(), stringValue)) </del> <ins> if strings.Contains(stringValue, ".") { allWarnings = append(allWarnings, fmt.Sprintf("%s: %q: prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers", fldPath.String(), stringValue)) } else { allWarnings = append(allWarnings, fmt.Sprintf("%s: %q: prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers", fldPath.String(), stringValue)) } </ins> } } return allWarnings

k8s component developers are using FQDNs as finalizers (e.g. or which triggers an error message that isn't particularly easy for an end user to understand: To a naive end user, a fully qualified domain name is a pretty safe thing that is unlikely to conflict with anything else (as long as it's only used by someone from within the domain that owns it, which is a reasonable assumption). If the finalizer has s inside it but no , the message should suggest that finalizers need a . Something like: Do whatever it is that triggered the warning in (this is probably simpler than installing cert-manager and asking for a certificate) introduced: <details</details<detailsGKE </details<details</details<details</details<details</details<details</details
/sig api-machinery
If thats ok I would like to work on that.
/assign
/triage accepted


Question: }, requestCount: 5, }, <ins> { name: "add taint to non-exist node", nodeHandler: &testutil.FakeNodeHandler{ Existing: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{ Name:            "node1", ResourceVersion: "1", }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "key1", Value: "value1", Effect: "NoSchedule"}, }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), }, nodeName:    "node2", taintsToAdd: []*v1.Taint{{Key: "key2", Value: "value2", Effect: "NoExecute"}}, expectedErr: apierrors.NewNotFound(schema.GroupResource{Resource: "nodes"}, "node2"), }, </ins> } for _, test := range tests { err := AddOrUpdateTaintOnNode(context.TODO(), test.nodeHandler, test.nodeName, test.taintsToAdd...) <ins> if test.expectedErr != nil { assert.Equal(t, test.expectedErr, err, "AddOrUpdateTaintOnNode get unexpected error") continue } </ins> assert.NoError(t, err, "%s: AddOrUpdateTaintOnNode() error = %v", test.name, err) node, _ := test.nodeHandler.Get(context.TODO(), test.nodeName, metav1.GetOptions{})

pkg/controller in I transformed that testcase into a fuzz driver and tested it with ]() It crash with "invalid memory address or nil pointer dereference" when some argument related data are fuzzed. Always No response No response Simplified PoC code is here: and it crashed as: /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /sig node /priority backlog This nil pointer is never reported by end users which means that before we call AddOrUpdateTaintOnNode, the node existence is checked. To make the code more stable, fixing it is valid. This would avoid future bugs if somewhere calls AddOrUpdateTaintOnNode without checking whether the node exists. I opened for this. /remove-kind failing-test (which means falling test ci )
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: return true, objv1, nil }) testClient.PrependReactor("update", "horizontalpodautoscalers", func(action core.Action) (handled bool, ret runtime.Object, err error) { <del> tc.Lock() defer tc.Unlock() </del> assert.Fail(t, "should not have attempted to update the HPA when nothing changed") // mark that we've processed this HPA tc.processed <- ""

What happened: The unit test for Horizontal Pod Autoscaler frequently deadlocked on the OpenShift CI system. What you expected to happen: The unit test should be robust and succeed without having to win a race condition to avoid deadlock. How to reproduce it (as minimally and precisely as possible): I'm not sure. Anything else we need to know?: Here are some logs of the problem happening: https://openshift-gce- The deadlocks seem to happen when the chan fills up while waiting for events. If that happens, there's a race where immediately after an event is read but before it can acquire the lock to verify the test, 2 event reactors are able to acquire the lock and write to the chan. The second will block forever since the chan is full and there's nobody to read from it. I've opened a PR with the fix. See
/sig autoscaling
/assign


Question: Subsystem:      SchedulerSubsystem, Name:           "schedule_attempts_total", Help:           "Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"result", "profile"}) e2eSchedulingLatency = metrics.NewHistogramVec( &metrics.HistogramOpts{ <ins> Subsystem:         SchedulerSubsystem, Name:              "e2e_scheduling_duration_seconds", DeprecatedVersion: "1.23.0", Help:              "E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.", Buckets:           metrics.ExponentialBuckets(0.001, 2, 15), StabilityLevel:    metrics.ALPHA, }, []string{"result", "profile"}) schedulingLatency = metrics.NewHistogramVec( &metrics.HistogramOpts{ </ins> Subsystem:      SchedulerSubsystem, <del> Name:           "e2e_scheduling_duration_seconds", Help:           "E2e scheduling latency in seconds (scheduling algorithm + binding)", </del> <ins> Name:           "scheduling_attempt_duration_seconds", Help:           "Scheduling attempt latency in seconds (scheduling algorithm + binding)", </ins> Buckets:        metrics.ExponentialBuckets(0.001, 2, 15), <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"result", "profile"}) SchedulingAlgorithmLatency = metrics.NewHistogram( &metrics.HistogramOpts{

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: return template }(), }, <ins> "good-parameters-apigroup": { template: func() *resource.ResourceClaimTemplate { template := testClaimTemplate(goodName, goodNS, goodClaimSpec) template.Spec.Spec.ParametersRef = &resource.ResourceClaimParametersReference{ APIGroup: goodAPIGroup, Kind:     "foo", Name:     "bar", } return template }(), }, "bad-parameters-apigroup": { wantFailures: field.ErrorList{field.Invalid(field.NewPath("spec", "spec", "parametersRef", "apiGroup"), badAPIGroup, "a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')")}, template: func() *resource.ResourceClaimTemplate { template := testClaimTemplate(goodName, goodNS, goodClaimSpec) template.Spec.Spec.ParametersRef = &resource.ResourceClaimParametersReference{ APIGroup: badAPIGroup, Kind:     "foo", Name:     "bar", } return template }(), }, </ins> "missing-parameters-kind": { wantFailures: field.ErrorList{field.Required(field.NewPath("spec", "spec", "parametersRef", "kind"), "")}, template: func() *resource.ResourceClaimTemplate {

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: framework.ExpectEqual(string(job.UID), statusUID, fmt.Sprintf("job.UID: %v expected to match statusUID: %v ", job.UID, statusUID)) }) <ins> /* Description: Attempt to create a suspended Job which MUST succeed. Attempt to patch the Job to include a new label which MUST succeed. The label MUST be found. Attempt to replace the Job to include a new annotation which MUST succeed. The annotation MUST be found. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. It MUST succeed at deleting a collection of jobs via a label selector. */ ginkgo.It("should manage the lifecycle of a job", func() { jobName := "e2e-" + utilrand.String(5) label := map[string]string{"e2e-job-label": jobName} labelSelector := labels.SelectorFromSet(label).String() ns := f.Namespace.Name jobClient := f.ClientSet.BatchV1().Jobs(ns) w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = labelSelector return jobClient.Watch(context.TODO(), options) }, } jobsList, err := jobClient.List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to list Job") ginkgo.By("Creating a suspended job") job := e2ejob.NewTestJob("succeed", jobName, v1.RestartPolicyNever, parallelism, completions, nil, backoffLimit) job.Labels = label job.Spec.Suspend = pointer.BoolPtr(true) job, err = e2ejob.CreateJob(f.ClientSet, ns, job) framework.ExpectNoError(err, "failed to create job in namespace: %s", ns) ginkgo.By("Patching the Job") payload := "{"metadata":{"labels":{"" + jobName + "":"patched"}}}" patchedJob, err := f.ClientSet.BatchV1().Jobs(ns).Patch(context.TODO(), jobName, types.StrategicMergePatchType, []byte(payload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch Job %s in namespace %s", jobName, ns) ginkgo.By("Watching for Job to be patched") c := watchEventConfig{ framework:           f, resourceVersion:     jobsList.ResourceVersion, w:                   w, jobName:             jobName, watchEvent:          watch.Modified, extJob:              patchedJob, updatedMetadataType: "label", updatedKey:          jobName, updatedValue:        "patched", } waitForJobEvent(c) framework.ExpectEqual(patchedJob.Labels[jobName], "patched", "Did not find job label for this job. Current labels: %v", patchedJob.Labels) ginkgo.By("Updating the job") var updatedJob *batchv1.Job err = retry.RetryOnConflict(retry.DefaultRetry, func() error { patchedJob, err = jobClient.Get(context.TODO(), jobName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get job %s", jobName) patchedJob.Spec.Suspend = pointer.BoolPtr(false) patchedJob.Annotations["updated"] = "true" updatedJob, err = e2ejob.UpdateJob(f.ClientSet, ns, patchedJob) return err }) framework.ExpectNoError(err, "failed to update job in namespace: %s", ns) ginkgo.By("Watching for Job to be updated") c = watchEventConfig{ framework:           f, resourceVersion:     patchedJob.ResourceVersion, w:                   w, jobName:             jobName, watchEvent:          watch.Modified, extJob:              updatedJob, updatedMetadataType: "annotation", updatedKey:          "updated", updatedValue:        "true", } waitForJobEvent(c) framework.ExpectEqual(updatedJob.Annotations["updated"], "true", "updated Job should have the applied annotation") framework.Logf("Found Job annotations: %#v", patchedJob.Annotations) ginkgo.By("Listing all Jobs with LabelSelector") jobs, err := f.ClientSet.BatchV1().Jobs("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "Failed to list job. %v", err) framework.ExpectEqual(len(jobs.Items), 1, "Failed to find job %v", jobName) testJob := jobs.Items[0] framework.Logf("Job: %v as labels: %v", testJob.Name, testJob.Labels) ginkgo.By("Waiting for job to complete") err = e2ejob.WaitForJobComplete(f.ClientSet, ns, jobName, completions) framework.ExpectNoError(err, "failed to ensure job completion in namespace: %s", ns) ginkgo.By("Delete a job collection with a labelselector") propagationPolicy := metav1.DeletePropagationBackground err = f.ClientSet.BatchV1().Jobs(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{PropagationPolicy: &propagationPolicy}, metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to delete job %s in namespace: %s", job.Name, ns) ginkgo.By("Watching for Job to be deleted") c = watchEventConfig{ framework:           f, resourceVersion:     updatedJob.ResourceVersion, w:                   w, jobName:             jobName, watchEvent:          watch.Deleted, extJob:              &testJob, updatedMetadataType: "label", updatedKey:          "e2e-job-label", updatedValue:        jobName, } waitForJobEvent(c) ginkgo.By("Relist jobs to confirm deletion") jobs, err = f.ClientSet.BatchV1().Jobs("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "Failed to list job. %v", err) framework.ExpectEqual(len(jobs.Items), 0, "Found job %v", jobName) }) </ins> }) <ins> // waitForJobEvent is used to track and log Job events. // As delivery of events is not actually guaranteed we // will not return an error if we miss the required event. func waitForJobEvent(config watchEventConfig) { f := config.framework ctx, cancel := context.WithTimeout(context.Background(), f.Timeouts.PodStartShort) defer cancel() _, err := watchtools.Until(ctx, config.resourceVersion, config.w, func(event watch.Event) (bool, error) { if job, ok := event.Object.(*batchv1.Job); ok { var key string switch config.updatedMetadataType { case "annotation": key = job.Annotations[config.updatedKey] case "label": key = job.Labels[config.updatedKey] } found := job.ObjectMeta.Name == config.extJob.ObjectMeta.Name && job.ObjectMeta.Namespace == f.Namespace.Name && key == config.updatedValue && event.Type == config.watchEvent if !found { framework.Logf("Event %v observed for Job %v in namespace %v with labels: %v and annotations: %v", event.Type, job.ObjectMeta.Name, job.ObjectMeta.Namespace, job.Labels, job.Annotations) return false, nil } framework.Logf("Event %v found for Job %v in namespace %v with labels: %v and annotations: %v", event.Type, job.ObjectMeta.Name, job.ObjectMeta.Namespace, job.Labels, job.Annotations) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) if err != nil { j, _ := f.ClientSet.BatchV1().Jobs(f.Namespace.Name).Get(context.TODO(), config.jobName, metav1.GetOptions{}) framework.Logf("We missed the %v event. Job details: %+v", config.watchEvent, j) } } </ins> // waitForJobFailure uses c to wait for up to timeout for the Job named jobName in namespace ns to fail. func waitForJobFailure(c clientset.Interface, ns, jobName string, timeout time.Duration, reason string) error { return wait.Poll(framework.Poll, timeout, func() (bool, error) {

[x] Test approval issue : [x] Test PR : - Needed an update on the handling of anotations 27 March 2022 [x] Two weeks soak end date : 6 April 2022 [x] Test promotion PR : [x] Update Test PR: 15 April 2022 [x] Two weeks soak end date : 4 May 2022 with start of 1.25 [x] Update Test Promotion: According to these APIsnoop queries, there are still four remaining Job endpoints which are untested. Using a number of existing e2e test practices a new has been created for this job lifecycle test. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for tracking e2e testgrid and promotion PR
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // LabelNodeRoleExcludeBalancer specifies that the node should be // exclude from load balancers created by a cloud provider. <del> LabelNodeRoleExcludeBalancer = "alpha.node.role.kubernetes.io/exclude-balancer" </del> <ins> LabelNodeRoleExcludeBalancer = "alpha.service-controller.kubernetes.io/exclude-balancer" </ins> ) type cachedService struct {

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: continue } // first failure <del> if containerStatus.State.Terminated != nil { </del> <ins> if containerStatus.State.Terminated != nil && !containerStatus.State.Terminated.FinishedAt.IsZero() { </ins> ts = containerStatus.State.Terminated.FinishedAt break }

If you create a pod with a restart policy of never, and you have a container that fails to start (e.g. you set its command to a nonexistent file), the Kubelet currently tries to restart it infinitely because the container never started successfully, and therefore never received an exit code. You end up with something like this from Docker: Is it worth modifying the code to consider a container that failed to start as actually failed, so the pod's restart policy can govern it? Originally reported at
+1. Also, I'm not sure why we would even apply a restart policy, since the container never started. , Andy Goldstein wrote:
One issue that comes up with marking it failed is if its governed by a Replication Controller, a consequence of marking it Failed is a proliferation of Pods being rapidly created and then failed. I know pod garbage collection would help that, but it would be good if we can update the dependent controller to recognize in common cases to back-off if pods keep failing because of failure to start scenarios. , Vish Kannan wrote:
I thought Kubelet already surfaces both creation and start failure from docker. Let me find the old issue I filed.
Here is the issue and the initial fix is PR: This is a bug we should fix anyway.
what is the output of kubectl describe pods? Are container at waiting state with proper error message from docker?
With respect to replication controller, see also . It should back off.
kubelet does surface the error (see the status below). The problem is that it doesn't categorize the container as dead/terminated, which affects the restart policy. A container would be considered "terminated" in kubelet if its state is not Running and the . In the case where a container failed to start, the FinishedAt time is zero (see example below), and the container state would be . This may cause kubelet to not even back off restarting at all (didn't check thoroughly), let alone complying to the restart policy. We should make sure kubelet categorize these containers properly.
Ok, I reproduced the issue easily. Here are the current situation: When container cannot start by docker due to various reasons, Kubelet marked the container as Waiting state with proper reason (RunContainerError) and detail message for docker, so Pod will stuck at Pending state.

I copied & pasted a PodStatus example below: When above situation occurs, Kubelet will retry in next syncLoop, which is the original issue reported here. Please note that I call this retry, instead of restart, which arguably align with our current , and do nothing. So that upstream layer can pick up such issue, for example, job controller in this case, and delete such bad pods. Or Kubelet can introspect ContainerStatus's reason: With restartPolicy never, Kubelet marks such Container as terminated with above proper reason. With restartPolicy onFailure, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. Retry / Restart will be backoff by the same backoff policy on the node. With restartPolicy Always, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. etry / Restart will be backoff by the same backoff policy on the node. For replication controlled pods, since we didn't mark Pod as Failed, there is no issue mentioned at
I think kubelet should not rely on the reason (which is best-effort), and simply stops relying solely on to categorize "terminated" containers.
what would you look at in addition to ?
in my example above, the container which failed to start had . To me, it's clear that there was a failed attempt trying to start the container. Even though it's finish time is zero, this container should be considered "terminated".
ah, good point! I hadn't noticed that ExitCode was nonzero.


Question: <del> v1.11.1-2 </del> <ins> v1.11.2-1 </ins>

go 1.11.2 is out 11/2 and this bug tracks upgrade to the latest version. Is it something we should absolutely needed in the 1.13 timeframe? are there any that should absolutely take? how long does the associated image updates and push take? If we chose to upgrade in 1.13, we should do it before code slush targeted for Friday 11/6. /assign can you please clarify the above ques and and help evaluate timing of the upgrade. /cc /sig release /sig testing /priority important-soon /kind cleanup
/cc I’ve already got a branch cooking for this, but waiting for rules_go support first. I think said in Slack that this will happen this week.
Ahh so is the earliest expected timeframe when we can upgrade falls into Code slush period for 1.13 (post 11/9?) Depending on when it lands we might want to see if its safe to wait for 1.13.1. are you aware of any critical fixes that make this compelling for 1.13 timeframe?
The risk for patch golang point releases is typically low (the minor releases, like 1.10 -1.11, are higher risk). I don't see anything that's critical or of known impact, but at the same time, I see compiler fixes and go/types package fixes. They are more subtle and could present issues later. I'd suggest that if we can tolerate getting it in, even if we're in slush, we should.
Ack that :) Plz ping this bug as soon as we are ready to build and push the upgraded images. we'll will make a final call depending on the timeframe and CI signal going in. thanks!


Question: // TODO: Make it work for all providers and distros. supportedProviders := []string{"gce", "aws", "vsphere"} if !ProviderIs(supportedProviders...) { <del> return fmt.Errorf("unsupported provider: %s, supported providers are: %v", TestContext.Provider, supportedProviders) </del> <ins> return fmt.Errorf("unsupported provider for RestartKubelet: %s, supported providers are: %v", TestContext.Provider, supportedProviders) </ins> } if ProviderIs("gce") && !NodeOSDistroIs("debian", "gci") { return fmt.Errorf("unsupported node OS distro: %s", TestContext.NodeOSDistro)

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by