Question: return clientset.CoreV1().Services(namespace).Create(service) } <del> func createLBService(clientset *fedclientset.Clientset, namespace, name string) (*v1.Service, error) { </del> <ins> func createLBService(clientset *fedclientset.Clientset, namespace, name string, clusters fedframework.ClusterSlice) (*v1.Service, error) { </ins> if clientset == nil || len(namespace) == 0 { return nil, fmt.Errorf("Internal error: invalid parameters passed to createService: clientset: %v, namespace: %v", clientset, namespace) }

In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


Question: "//:all-srcs", ], outs = ["stable-metrics-list.yaml"], <del> cmd = "./$(locations :instrumentation) $(locations //:all-srcs) > $@", </del> <ins> cmd = "for loc in $(locations //:all-srcs); do echo $$loc; done | ./$(locations :instrumentation) - > $@", </ins> message = "Listing all stable metrics.", tools = [":instrumentation"], )

/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM


Question: "resources but no selectors": { args: []string{"pods", "app=bar"}, errFn: func(err error) bool { <del> return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") </del> <ins> return strings.Contains(err.Error(), "resource(s) were provided, but no name was specified") </ins> }, }, "multiple resources but no selectors": { args: []string{"pods,deployments", "app=bar"}, errFn: func(err error) bool { <del> return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") </del> <ins> return strings.Contains(err.Error(), "resource(s) were provided, but no name was specified") </ins> }, }, }

What happened: Kubectl patch sub-command shows wrong error help message if an input is wrong. The error message says and when I try to use flag it says . What you expected to happen: I expected either the first error message to not have or should have a flag . How to reproduce it (as minimally and precisely as possible): If you run the following command without a node name you get this error: As the above error said you can specify , you specify that. This fails with another error saying is unknown flag: And in the help there is no flag: Environment: Kubernetes version (use ):
/sig cli
I think we need to decide whether to add flag --all and --selector to kubectl patch. After the decision, i can help.
/assign
on my queue, will look tomorrow :D
/triage accepted Probably just need to update error message here.
oh sorry, missed that one. Will look now
/assign
OK, so we decided to correct the error message per the PR right? :)
as you're dealing with this PR, I'll left you as assignee and follow the PR review :) /unassign


Question: return errors.Wrapf(err, "cannot get Node %q", nodeName) } for _, cond := range node.Status.Conditions { <del> if cond.Type == v1.NodeReady { </del> <ins> if cond.Type == v1.NodeReady && cond.Status == v1.ConditionTrue { </ins> return errors.Errorf("a Node with name %q and status %q already exists in the cluster. "+ "You must delete the existing Node or change the name of this new joining Node", nodeName, v1.NodeReady) }

What happened: After upgrading to v1.18.0 not possible to join existing node into cluster: <details<summaryFull log</summary<preW0326 11:57:39. ] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set. I0326 11:57:39. ] [preflight] found NodeName empty; using OS hostname as NodeName I0326 11:57:39. ] detected and using CRI socket: [preflight] Running pre-flight checks I0326 11:57:39. ] [preflight] Running general checks I0326 11:57:39. ] validating the existence and emptiness of directory /etc/kubernetes/manifests I0326 11:57:39. ] validating the existence of file I0326 11:57:39. ] validating the existence of file /etc/kubernetes/bootstrap- I0326 11:57:39. ] validating the container runtime I0326 11:57:39. ] validating if the service is enabled and active I0326 11:57:39. ] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables I0326 11:57:39. ] validating the contents of file /proc/sys/net/ipv4/ipforward I0326 11:57:39. ] validating whether swap is enabled or not I0326 11:57:39. ] validating the presence of executable conntrack I0326 11:57:39. ] validating the presence of executable ip I0326 11:57:39. ] validating the presence of executable iptables I0326 11:57:39. ] validating the presence of executable mount I0326 11:57:39. ] validating the presence of executable nsenter I0326 11:57:39. ] validating the presence of executable ebtables I0326 11:57:39. ] validating the presence of executable ethtool I0326 11:57:39. ] validating the presence of executable socat I0326 11:57:39. ]

validating the presence of executable tc I0326 11:57:39. ] validating the presence of executable touch I0326 11:57:39. ] running all checks I0326 11:57:39. ] checking whether the given node name is reachable using net.LookupHost I0326 11:57:39. ] validating kubelet version I0326 11:57:39. ] validating if the service is enabled and active I0326 11:57:39. ] validating availability of port I0326 11:57:39. ] validating the existence of file I0326 11:57:39. ] validating if the connectivity type is via proxy or direct I0326 11:57:39. ] [preflight] Discovering cluster-info I0326 11:57:39. ] [discovery] Created cluster-info discovery client, requesting info from "10.28.36.157:8443" I0326 11:57:39. ] [discovery] Requesting info from "10.28.36.157:8443" again to validate TLS against the pinned public key I0326 11:57:39. ] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.28.36.157:8443" I0326 11:57:39. ] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process I0326 11:57:39. ] [preflight] Fetching init configuration I0326 11:57:39. ] [preflight] Retrieving KubeConfig objects [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' I0326 11:57:39. ] Looking for default routes with IPv4 addresses I0326 11:57:39. ] Default route transits interface "vmbr0" I0326 11:57:39. ] Interface vmbr0 is up I0326 11:57:39. ]

Interface "vmbr0" has 3 addresses :[10.28.36.164/16 2a02:2b88:1:2:9657:a5ff:fed3:b6f2/64 fe80::9657:a5ff:fed3:b6f2/64]. I0326 11:57:39. ] Checking addr 10.28.36.164/16. I0326 11:57:39. ] IP found 10.28.36.164 I0326 11:57:39. ] Found valid IPv4 address 10.28.36.164 for interface "vmbr0". I0326 11:57:39. ] Found active IP 10.28.36.164 I0326 11:57:39. ] [preflight] Running configuration dependant checks I0326 11:57:39. ] [download-certs] Skipping certs download I0326 11:57:39. ] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap- I0326 11:57:39. ] [kubelet-start] writing CA certificate at I0326 11:57:39. ] [kubelet-start] Checking for an existing Node in the cluster with name "m1c4" and status "Ready" a Node with name "m1c4" and status "Ready" already exists in the cluster.

You must delete the existing Node or change the name of this new joining Node /workspace/anago-v1.18.0- (Runner).Run.func1 /workspace/anago-v1.18.0- (Runner).visitAll /workspace/anago-v1.18.0- (Runner).Run /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- (Command).execute /workspace/anago-v1.18.0- (Command).ExecuteC /workspace/anago-v1.18.0- (Command).Execute /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- /usr/local/go/src/runtime/asmamd64.s:1357 error execution phase kubelet-start (Runner).Run.func1 /workspace/anago-v1.18.0- (Runner).visitAll /workspace/anago-v1.18.0- (Runner).Run /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- (Command).execute /workspace/anago-v1.18.0- (Command).ExecuteC /workspace/anago-v1.18.0- (Command).Execute /workspace/anago-v1.18.0- /workspace/anago-v1.18.0- /usr/local/go/src/runtime/asmamd64.s:1357 </pre</detailsThe problem is that the check introduced in violates the existing diskless node approach. Eg. we are having cluster of hundreds servers which are booding over PXE and have no any local storage. After the boot they just running command to connect to the cluster, they can be rebooted ay any time, and after reboot they will run join procedure again. Thus node object in cluster may already have some settings, labels and annotation for this node. After the joining the node just reuses them, this is working as a charm. What you expected to happen: Existing node successful joined How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Currently if you try to join the node you will face with , but I created cluster role to check this chage.

Environment: Kubernetes version: Cloud provider or hardware configuration: Bare Metal OS: Kernel: Install tools: Network plugin and version Others:
/sig cluster-lifecycle /area kubeadm
we have to be clear that this approach was never officially supported by kubeadm. simply restarting a node machine would make the kubelet and the CR bring that node back up. but resetting the node and then re-joining without deleting the node object is problematic. by preventing a node with the same name from joining the cluster we are solving a security issue, as otherwise an attacker knowing a bootstrap token would be able to bring your control-plane down. ideally, your steps should now be: /priority awaiting-more-evidence
/assign
OK, but this opportunity is supported by kubelet and api-server itself, it is working fine and I don't see the reason to stop using it. We are using this approach since v1.9 and never with the problems described in , every rejoin operation of old node is just reuse name and parameters of existing resource with no any problems. This is not fully truth, because: He can just use modified kubeadm binary, or other deployment tool. And now he can also list all the nodes in cluster to make a choose which one he want to hack Bootstrap token could be generated automatically every hour and for each new boot This precaution is useless for spheres where Kubernetes is used in closed perimeter just for convenient utilizing a compute power of physical servers, eg scientific where you need just convenient tool to perform calculating and no security.
We are using this approach since v1.9 and never with the problems described in , every rejoin operation of old node is just reuse name and parameters of existing resource with no any problems. i've seen in the wild a number of times and users are shooting them self in the foot causing outage of the control-plane creating tickets for us. this is absolutely true i should have said "partially" as it needs a server side fix as well. it only eliminates the issue for users that do not recompile the source code (for this particular version, for newer version they still need the RBAC) or users that do it by accident but the malicious bootstrap can only succeed if the user knows a token. they can still shoot themself in the foot by having two nodes with the same name. so i think it has some value.

using kubernetes without security is not possible - e.g. disabling certificates completely.
OK, but I still do not understand how is it possible to create two nodes with the same name, is there any steps to reproduce? I was trying but Kuberentes API is not allowing that: The same as flag, however it is not breaking existing functionality, but still alerting the user about a possible problem.
creating the Node object directly will not work. the TLS bootstrap in the kubelet for matching node names (hostnames) can still trigger the race condition in etcd/apiserver
if we will restrict rejoining only for new nodes, say, whose age is 1 minute, would it solve the problem with the race condition in etcd?
Or even better: to not restrict the join if node having status.
not sure i understand. if you join a node named "foo" to a cluster that already has "foo", the new "foo" will remain NotReady but the existing "foo" will also become NotReady due to the crash loop of control-plane components. not sure how age matters here. are you suggesting the Node having status NodeReady but with condition Unknown: what would put it in this condition? my understanding is that you are trying to rejoin nodes while their status is still "Ready", so this line breaks this for you: would changing and possibly adjusting of the KCM to lower values and waiting for the nodes to become NotReady before joining solve the issue for you?
Unfortunately intentionally or accidentally, but this check does not work: In #diff-3c566a0c1e3b213a34b39503abf4fc11R150, you're checking just existence of condition with but you also have to check the of this condition
This variant is working fine:
it did work in my tests. i guess, it would not work for the case where the condition is still present but not True, so your PR makes sense.
but that does not help you with your use case, right?
This change is fine for me, because I need to rejoin nodes only after reboot. Usually it takes more than 30 seconds (it's a time when node is coming from to state.)
so by the time you reboot the NodeReady condition now has a status instead of ? does affect this? it's default is 40s.
you might want to lower that value.

also see my comment about "release note" on the PR.
You're right it is 40 seconds:
ok, i will send a cherry pick for the PR to 1.18


Question: namespace, err := namer.Namespace(req) if err != nil { <del> notFound(w, req.Request) </del> <ins> errorJSON(err, codec, w) </ins> return } ctx := ctxFn(req)

e.g. REST.Update should never return an arbitrary error value. Maybe we should change the return code to something stricter than generic error.
/cc
FYI: is for remaining cases where the API doesn't return JSON/Status errors.
Re-opening this until we
I think etcd errors require more abstraction, also:
There are some admission control plugins that did not always follow that practice which I will resolve as part of
Are there any known instances of this remaining?
None that I know of.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Unless disabled, the tests will prepull all images when running. They do so using the . I think there's the opportunity to improve the runtime of the test suite by prepulling the images in parallel. Why is this needed: Its always nicer to have faster tests :) Specifically for users running locally, it's nice to have the image prepull occur as quick as possible. That being said, we should also be cognizant of any tradeoffs associated with prepulling the images in parallel. If the code is substantially more complex, we would need a large improvement in test speed to justify it. I'm hopeful this discussion could be had once we have a proof of concept diff (which I think should be fairly minimal effort to create).
/sig node
I think this could be a good first issue for someone looking to get involved with contributing to sig-node :) Please post if you plan on working on it, so that there aren't multiple people tackling it.
I'd be interested! I'm looking for a first issue and think this would be a good one
Wonderful! Want to reach out to me on Kubernetes slack? I'm on there too :)
:wave: hiya ! Are you still thinking you could be interested in working on this? Lmk if so, and we can chat about next steps? If not, perhaps I'll open it back up to a newcomer looking for a first commit?
I'm going to open this back up for someone looking for a first commit. Lmk if you're still interest in getting sig-node, and happy to find an issue for you :)
Opening this back up for folks :) I'm happy to mentor whoever picks this up if that'd be helpful :) Ping me on k8s-slack, where I'm also
Can you assign this to meI'd be interested!
/assign
Awesome - feel free to reach out on k8s-slack if you have questions, or if you already have a plan, feel free to tag me in the pull request. Thanks :)
/lifecycle active /priority important-longterm


Question: output_message=$(kubectl diff -f hack/testdata/pod-changed.yaml || test $? -eq 1) kube::test::if_has_string "${output_message}" 'k8s.gcr.io/pause:3.0' <ins> # Test found diff with server-side apply kubectl apply -f hack/testdata/pod.yaml output_message=$(kubectl diff -f hack/testdata/pod-changed.yaml --server-side --force-conflicts || test $? -eq 1) kube::test::if_has_string "${output_message}" 'k8s.gcr.io/pause:3.0' </ins> # Test that we have a return code bigger than 1 if there is an error when diffing kubectl diff -f hack/testdata/invalid-pod.yaml || test $? -gt 1

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <ins> "k8s.io/api/core/v1" </ins> "k8s.io/klog" <del> "k8s.io/api/core/v1" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.

+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs?

From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-


Question: // Addf adds additional data to be logged with this request. func (rl *respLogger) Addf(format string, data ...interface{}) { <ins> rl.mutex.Lock() defer rl.mutex.Unlock() </ins> rl.addedInfo.WriteString("n") rl.addedInfo.WriteString(fmt.Sprintf(format, data...)) }

Apiserver crashed with stacktrace: Not to crash I was able to reproduce it by "slowing down" deleting headers in Authentication filter (): And making requests (around 5-10 requests to trigger this race condition): Crash was caused by race condition - one goroutine was modifying headers of requests (for example authentication filter). The other goroutine was trying to get UserAgent from request, which is also taken from request's headers map. This causes unrecoverable panic. <detailsAll versions, tested on 1.21 </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /priority important-soon
/triage accepted
Reopening - since we need to cherrypick it to previous releasses.
- friendly ping on opening cherrypicks
I've opened two cherry-picks:
One more for 1.20:
All merged - thanks!


Question: } for i, test := range tests { <del> findResult := findRule(test.existingRule, test.curRule) </del> <ins> findResult := findRule(test.existingRule, test.curRule, true) </ins> assert.Equal(t, test.expected, findResult, fmt.Sprintf("TestCase[%d]: %s", i, test.msg)) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the Idle Timeout annotation was set to "3" which isn't a supported case, kube-controller-manager is stuck in deleting Azure LB. New service with the same name couldn't be created again even without the annotation. What you expected to happen: New service should be created successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug
/assign Since I'm seeing this on some of our clusters
/priority important-soon
Thanks
- The problem here seems that when we want to call (when processing a delete event) it reconciles the loadbalancer which in turn tries to get the idleTimeout, resulting in an error. one of the strategies is to still construct our diff in case of a delete and exercise the code path at the end. This basically translate into ignoring the error when getting idleTimeout and excluding it from finding if the rule is dirty or not when we are . toughts ? Generally speaking, whenever we update a loadbalancer with unsupported values on annotations, ensuring load balancer will fail resulting in an un-cleaned LB, this is a cross-provider issue. /sig cloud-provider cc
cc
lgtm. the delete operation should continue to do cleanup job even some wrong options are provided from users.


Question: // ValidateOwnerReferences validates that a set of owner references are correctly defined. func ValidateOwnerReferences(ownerReferences []metav1.OwnerReference, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{} <del> controllerName := "" </del> <ins> firstControllerName := "" </ins> for _, ref := range ownerReferences { allErrs = append(allErrs, validateOwnerReference(ref, fldPath)...) if ref.Controller != nil && *ref.Controller { <del> if controllerName != "" { </del> <ins> curControllerName := ref.Kind + "/" + ref.Name if firstControllerName != "" { </ins> allErrs = append(allErrs, field.Invalid(fldPath, ownerReferences, <del> fmt.Sprintf("Only one reference can have Controller set to true. Found "true" in references for %v and %v", controllerName, ref.Name))) </del> <ins> fmt.Sprintf("Only one reference can have Controller set to true. Found "true" in references for %v and %v", firstControllerName, curControllerName))) </ins> } else { <del> controllerName = ref.Name </del> <ins> firstControllerName = curControllerName </ins> } } }

When there are more than one controller set to true within ownerRef, the validation would return err msg like this: It is confusing to see two same 'foo' controller name. To make the err msg more accurately, like this: Set an object with two , and both of the Controller set true, then apply to the apiserver will got this error.
/sig api-machinery
/triage accepted


Question: //     performed in the host's mount namespace do not propagate out to the //     bind-mount in this docker version. // 2.  The host's root filesystem must be available at /rootfs <del> // 3.  The nsenter binary must be at /nsenter in the container's filesystem. </del> <ins> // 3.  The nsenter binary must be on the Kubelet process' PATH in the container's //     filesystem. </ins> // 4.  The Kubelet process must have CAP_SYS_ADMIN (required by nsenter); at //     the present, this effectively means that the kubelet is running in a //     privileged container. <ins> // 5.  The volume path used by the Kubelet must be the same inside and outside //     the container and be writable by the container (to initialize volume) //     contents. TODO: remove this requirement. // 6.  The host image must have mount, findmnt, and umount binaries in /bin, //     /usr/sbin, or /usr/bin </ins> // // For more information about mount propagation modes, see: //   https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt <del> type NsenterMounter struct{} </del> <ins> type NsenterMounter struct { // a map of commands to their paths on the host filesystem paths map[string]string } func NewNsenterMounter() *NsenterMounter { m := &NsenterMounter{ paths: map[string]string{ "mount":   "", "findmnt": "", "umount":  "", }, } // search for the mount command in other locations besides /usr/bin for binary := range m.paths { // default to root m.paths[binary] = filepath.Join("/", binary) for _, path := range []string{"/bin", "/usr/sbin", "/usr/bin"} { binPath := filepath.Join(hostRootFsPath, path, binary) if _, err := os.Stat(binPath); err != nil { continue } m.paths[binary] = binPath break } // TODO: error, so that the kubelet can stop if the mounts don't exist } return m } </ins> // NsenterMounter implements mount.Interface var _ = Interface(&NsenterMounter{})

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.


Question: func TestHealthy(t *testing.T) { testPleg := newTestGenericPLEG() <ins> // pleg should initially be unhealthy </ins> pleg, _, clock := testPleg.pleg, testPleg.runtime, testPleg.clock ok, _ := pleg.Healthy() <del> assert.True(t, ok, "pleg should be healthy") </del> <ins> assert.False(t, ok, "pleg should be unhealthy") </ins> // Advance the clock without any relisting. clock.Step(time.Minute * 10)

When the kubelet first starts the folllowing error is emitted to the logs This is because the relist has not yet run which results in a duration that is just the time since the zero time instant () xref /sig node
/assign
/approve /lgtm
help if i approve the pr and not the issue ;-)


Question: } var err error <del> var output string podName := g.kubeControllerManager g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, podName, 0); readyErr != nil { err = fmt.Errorf("error waiting for controller manager pod to be ready: %w", readyErr) return } var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeControllerManager, metav1.NamespaceSystem, kubeControllerManagerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err = fmt.Errorf("error waiting for controller manager pod to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return </del> <ins> g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, g.kubeControllerManager, 0); readyErr != nil { err = fmt.Errorf("error waiting for kube-controller-manager pod to be ready: %w", readyErr) </ins> } }) if err != nil { return ControllerManagerMetrics{}, err } <ins> var output string var lastMetricsFetchErr error if metricsWaitErr := wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output, lastMetricsFetchErr = g.getSecureMetricsFromPod(g.kubeControllerManager, metav1.NamespaceSystem, kubeControllerManagerPort) return lastMetricsFetchErr == nil, nil }); metricsWaitErr != nil { err := fmt.Errorf("error waiting for kube-controller-manager to expose metrics: %v; %v", metricsWaitErr, lastMetricsFetchErr) return ControllerManagerMetrics{}, err } </ins> return parseControllerManagerMetrics(output) } // GrabFromSnapshotController returns metrics from controller manager func (g *Grabber) GrabFromSnapshotController(podName string, port int) (SnapshotControllerMetrics, error) { if !g.grabFromSnapshotController { <del> return SnapshotControllerMetrics{}, fmt.Errorf("snapshot controller: %w", MetricsGrabbingDisabledError) </del> <ins> return SnapshotControllerMetrics{}, fmt.Errorf("volume-snapshot-controller: %w", MetricsGrabbingDisabledError) </ins> } // Use overrides if provided via test config flags. <del> // Otherwise, use the default snapshot controller pod name and port. </del> <ins> // Otherwise, use the default volume-snapshot-controller pod name and port. </ins> if podName == "" { podName = g.snapshotController }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pvc count metrics for pvc controller after creating pvc only Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create unbound pv count metrics for pvc controller after creating pv only Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning and attach/detach Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create total pv count metrics for with plugin and volume mode labels after creating pv Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics should create prometheus metrics for volume provisioning errors [Slow] Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create none metrics for pvc controller before creating any PV or PVC Kubernetes e2e suite: [sig-storage] [Serial] Volume metrics PVController should create bound pv/pvc count metrics for pvc controller after creating both pv and pvc Test 2021-05-11 19:20:26 +0000 UTC /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think that this will be fixed by cc
Now we just need to convince someone to merge that PR...
The PR mentioned above was merged 10 days ago, but it didn't fix the failing tests. I checked the recent failures, and I didn't see anything failing exactly with the error reported ().

However, the failures look very similar, there are many timeouts when getting the metrics, like: Most of Volume metrics PVController tests are failing with this error: has some random passes, but usually fails with: also has random passes, but usually fails: fails fetching c-m metrics: also fails to find one metric: These all look related :) There are also some cleanup failures in the same job, but I don't think they're related, they started more recently.
If I read correctly, this means that the test doesn't get the metrics that it expects. I'm not entirely sure, but I think if it ran into an actual error, it would fail differently. Can you check whether there's any mention of "Error fetching controller-manager metrics"? This shows that metrics grabbing worked. It found some metrics data, it's just not the expected data. Not related to metrics? Another variant of "didn't get expected metrics".
Nope, there is nothing like that in . Ah, yes, this one doesn't look directly related, and it's different SIG. We should track it in a separate issue probably, I'll check it. I opened another issue for tracking unmounting failures in the same job:
Yes. There was that error message 29 days ago (06/14/2021) when I created this issue. Here is the But now, the error message is not exactly the same but this new one fails at the same step (Getting plugin name).
I think there's enough evidence to claim (until proven otherwise, of course :sweat_smile: ) that metrics grabbing now works. Why the metrics data is not what the tests expect I don't know - someone familiar with these tests and how they apply to Windows would have to check that.
Thanks for the suggestion I'll look for someone in the community who can help us to test this and update the tests if It is necessary.
has a meeting in ~30 minutes and I politely asked to review failures :)
FTR - those tests aren't related to Windows (the windows tests might have their own issues).
BTW - shouldn't we add this to 1.22 milestone?

/milestone v1.22
/assign
might be related
timed out with: I saw that the failure is in the call to , in the test there's a beforeEach hook where metrics are fetched with a function wrapped inside a successfully, However the test attempts to get metrics again later using the same function but because it's wrapped with the output is an empty string next time. Do you know why the function that fetches metrics is wrapped with ? cc
The sync.Once function are used for waiting for the corresponding pods once, I think it is a mistake, and the logic to obtain metrics should be out of the sync.Once function
Thanks for chiming in indeed once I removed the wrapping all of the tests passed in
/reopen I'll check the results in CI and also cherrypick to 1.22, then we should be able to close this issue.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All of the storage metric related failures are green now, the remaining storage tests that are failing are tracked in
thanks let's close this then! /close
Closing this issue. <detailsIn response to : thanks let's close this then! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return err } if err := windows.AssignProcessToJobObject(job, windows.CurrentProcess()); err != nil { <del> return errors.Wrap(err, "windows.AssignProcessToJobObject failed") </del> <ins> return fmt.Errorf("windows.AssignProcessToJobObject failed: %w", err) </ins> } if windowsService {

Why is this needed? The functionality that the package provides is already present in golang's builtin errors (using in ) and the github repo is now read-only. /kind cleanup /priority backlog /good-first-issue Notes to people that want to collaborate Avoid creating a single PR for the entire repo. Create multiple PRs, each targeting a specific SIG. Do not submit PRs for
/area code-organization
/kind cleanup
this is only partially true, e.g. %w does not provide stack traces
/assign
I see. Should we close this issue then?
I think it depends on how it's being used. If the traces are not the goal then we can do a simple substitution. It's probably best to drop read only dependencies ASAP but it's worth noting that if we depended on the full functionality in any way we'll need a more complex replacement (KIND did/does, not sure about k/k)
k/k git log suggests that some cleanup has already been done in the past (maybe in other commits too): This issue triggered 3 PRs and 2 of them are merged. there does not seem to be calls to so most cases it looks it's just wrapping errors.
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
I see. Added a note.
is it still open ?
should be replaced with the standard library for the following reasons: archived library with no maintenance or security updates the go community used pkg/errors to introduce eventually into the standard library hence the approach that was selected is more idiomatic stack traces are hardly used in error wrapping and are not a reason not to refactor
On place left in Except the k/k repo, as is archived, some other repos may need an update.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is it necessary to remove "" from every location? I noticed it in many places. Also, do I need to create separate PRs for each file where I remove it?
/assign
From kuberentes/kubernetes repo view, only two locations of kubectl code should be updated. (Do not submit PRs for ) see For other repos, we should follow up until the dependencies' repo is updated. I will track them in
/assign
/assign
to remove dependency you can use the standard Go error package or creating your own handling package, this could contain custom functions for generating and handling errors.
let me work on this issue now please /assign
We are in code freeze. I suggest waiting for a couple of weeks.
Hello, is this still up?
Let's get this going
/assign
Just opened a PR for this issue
/assign
Update the remaining todo items.
/sig architecture umbrella / code org
/assign
Hey Are you still working on this one? If not- would you mind if I take a stab at it? Thanks!
Is this issue really still open? When I do I only get hits for cmd/kubeadm. Am I missing something?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // RunAnnotate does the work <del> func (o AnnotateOptions) RunAnnotate() error { </del> <ins> func (o AnnotateOptions) RunAnnotate(f *cmdutil.Factory) error { </ins> r := o.builder.Do() if err := r.Err(); err != nil { return err } return r.Visit(func(info *resource.Info, err error) error { if err != nil { return err } <del> _, uErr := cmdutil.UpdateObject(info, func(obj runtime.Object) error { err := o.updateAnnotations(obj) if err != nil { return err } return nil }) if uErr != nil { return uErr </del> <ins> name, namespace, obj := info.Name, info.Namespace, info.Object oldData, err := json.Marshal(obj) if err != nil { return err } if err := o.updateAnnotations(obj); err != nil { return err } newData, err := json.Marshal(obj) if err != nil { return err </ins> } <del> return nil </del> <ins> patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, obj) if err != nil { return err } mapping := info.ResourceMapping() client, err := f.RESTClient(mapping) if err != nil { return err } helper := resource.NewHelper(client, mapping) _, err = helper.Patch(namespace, name, api.StrategicMergePatchType, patchBytes) return err </ins> }) }

and work in this pattern: first issuing a "GET" to the API server to retrieve the latest object, then adding the label/annotation to the object, and finally issuing an "PUT" with the modified object. The request will fail if the object stored in the server has been updated between the "GET" and "PUT", even if the update has no conflict with the label/annotate request. In such cases, instead of returning an error, should retry without disturbing the end user. There are two ways to implement this: in kubectl: simply retry the label/annotate process several times if the server rejected the request because the object has been updated. The only exception is when the is set, then kubectl shouldn't retry. the strategicMergePatch instead of calling Update at client-side. PATCH is more robust because PATCH does the "get, modify, update" in the server-side, so there is less chance to have a conflict, and PATCH will retry at the server-side (see ). We need to make some adaptions though: kubectl annotate/label has the , when it's false, kubectl should retrieve the latest object to confirm there is no existing annotation/label of the same key, and then proceed with issuing the patch. if the is set, [UPDATE:] we just need to specify the resourVersion in the patch, then it will be treated as a precondition by the UpdateResource() handler in the API server. I plan to take the first approach as an immediate fix.
Immediate need is to fix the flaky e2e test AIUI.
e2e? do we have an issue out there? is an immediate fix to the flake in test-
If this is for 1.1, it needs to be P0 at this point. Should it be?
Fwiw I use annotations both from cmd line as an admin and from scripts, to experiment with loadbalancer settings. In the case of the former, I've never really had a collision, and in the case of the latter my scripts know how to retry like we retry all other updates.
Thank you When used in scripts, the annotate and label commands have collisions frequently, e.g. in the test-, see . IMO, we shouldn't require users to retry in their scripts if we know there is a simple way to avoid the collisions. Anyway, and are sent to fix this.


Question: createTagSteps        = 9 ) <ins> // awsTagNameMasterRoles is a set of well-known AWS tag names that indicate the instance is a master // The major consequence is that it is then not considered for AWS zone discovery for dynamic volume creation. var awsTagNameMasterRoles = sets.NewString("kubernetes.io/role/master", "k8s.io/role/master") </ins> // Maps from backend protocol to ELB protocol var backendProtocolMapping = map[string]string{ "https": "https",

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Install tools: default install What happened: AWS EBS backed PersitantVolumeClaim created a volume in an AZ where there was a master but no nodes. Making it unusable. What you expected to happen: Zones with master and no nodes shouldn't be used to create the EBS Volume How to reproduce it (as minimally and precisely as possible): AWS setup across 3 AZs in one region Master in three AZs Nodes in 2 AZs (so one AZ has a Master but no Nodes) Create a Storage class to use EBS without specifying AZCreate PVC against StorageClassPVC can create a volume in an availability zone with no nodes Anything else do we need to know: Mentioned on Slack channel #sig-aws with who asked me to raise this
, Mengqi Yu wrote:
Running Kubernetes master in a zone that has no running nodes is quite strange and I don't think this is a typical case we should support. AWS cloud provider lists all machines in the cluster and chooses a random zone that has at least one running instance. It seems it does not distinguish master/node machines. IMO, GCE has the same problem, it lists all zones in a region where the master runs and it does not distinguish masters and nodes either. We can either introduce a new AWS instance tag with master/node role that is running on the instance or we can move the zone decision into volume plugin itself, where we have a kubeclient and we can filter nodes relatively easily, however we will depend on an admission controller that must add a zone label to nodes. thoughts? As a third option, we could rely on kubernetes admin to fill all zones into StorageClass in this weird case. Right now we support only one zone in "zone" parameter, we could allow a list:
just to add to this, the reason why it's a deal is I actually don't have full control over where the nodes run, as they're in an autoscale group across all AZs. So you could have master in AZ a and nodes in b and c just because that's what the auto scale group picked.
I just ran into this, however, I only have nodes / masters in 3 zones, but the PV was created in a 4th zone.

So right now I can't even spin up a node in the last AZ since in my infrastructure, I'm not supporting it.
I think it may be more than just where there are or not nodes vs masters in an AZ. As an additional use case, consider pods that must run on a "dedicated" tenancy node in AWS, (which have a baseline cost per-zone) for compliance. Since masters are supposed to be an odd number (i.e. 3 zones) it would make sense to have 3 instead of 2. But for a pod that has to be scheduled on a "dedicated" tenancy node, there may be only 2 zones with nodes that are "dedicated" due to said cost. Even if there are nodes in all zones, a pod may need to be scheduled on a specific node or subset thereof. More information about node selection: More information about why this specific case would be necessary here: If PVs are used by pods, and pods must be able to access them from whatever nodes they are constrained/scheduled to, I think whatever solution is implemented needs to account for that.
do you have any other non-Kubernetes instances in us-east-1b in your AWS account? Did you try to tag your AWS instances that run Kubernetes with tag - all AWS instances in one Kubernetes cluster must share the same tag value, so you can run multiple Kubernetes clusters in one AWS account. This tag is not documented, see .
There is another non-k8s instance running in us-east-1b, but it's not related (created via breanstalk). I labels to all my nodes / controller as you sugessted and still it created the PV in . I may bail on Statefulset for now and manage my pv's by hand since I'm building out an operator and it wont be that painful, but would like to see if this could be addressed. I'm also willing to help out if you can help point into the right places this logic lives. I'll also try and dig in and see if I can find it myself.
It would be ideal to have the StorageClass accept an expression or wildcard zone under the aws provisioner. Ideally I shouldn't have to care where my pods are running and where the EBS are created.
We've a PR that adds "zones:" with list of zones instead of single "zone:". No wildcards though.

I just hit this as well. I think moving the "zone" decision closer to the provisioning decision would be best. Perhaps wait until a PVC is consumed by a pod before you create it in a zone. That way corner cases like this are not a problem.
Just want to check whether PR is good enough to solve the issue raised here?
, is right. Another use case is that if pod consumes 1 PVC, associated PVs might be created in different zones, therefore pod becomes unschedulable, as there is no node which belongs 1 AZ. Zone decision when provisioning PV must be made as late as possible, while currently it seems to be a very first thing to do.
I don't think it can be closed yet, reasons are: those magical tags are not documented anywhere, how people or going to know to use them? can't it continue to create volume in a zone where pod with a given PVC can never be scheduled (lets say it has node affinity configured to a node in a particular AZ)?
Would it make sense to say that the dynamic creation of a volume should happen only once a PVC has been attached to a pod which has been successfully scheduled to a node? Then the av_zone of the volume should match that of the selected node?
Yes I would agree with that statement. The PVC should be created when we know the node and AZ it is a apart of.
Just to add another use case to this - I just spun up a new cluster in AWS. Masters spanning , , and . After creating the ASG for nodes, I learned that AWS is out of instance capacity in for that particular instance type. Now I currently don't have nodes in (until capacity is restored and I can move an instance over). The reason this is different is because, in this case, I don't want to tell k8s to only use a subset of AZs. I want it to deploy across all AZs that are available. It just so happens that at this moment in time not all AZs are available. suggestion above would work, I believe. Schedule the node first, then provision a volume. That might cause other problems, though.
I just ran into this today - PVC created in eu-west-1c, but the only node with memory to to run the pod was in eu-west-1b.

This was a new pod and a new volume claim. Can we have this re-opened at least please?
We had to ultimately get rid of all persistence volume claims on AWS because of this. For really large clusters with lots of reserved capacity in all availability zones that woulnd't be such a big deal, but I'd image most companies are trying to optimize their cost and don't have that many node pre-provisioned just for the sake of it. Still looking for a solution which is easy to configure and AZ-failure friendly
Right now, there isn't really. You have a few options: EFS - it is regional, not zonal. You have to be OK working with NFS as the protocol, and potential performance implications. To be fair, NFS has gotten quote good, I know companies using it to store critical data. I am not an NFS expert, so don't ask my opinions here... Commercial solutions - Portworx and StorageOS both provide local storage, backed by local disk or EBS, but replicated at the app level (their app) between nodes, across zones, so you don't depend on zonal resources (like EBS). DIY - have fun. Ceph, some other solutions. For one use case with which I am involved, we are living with it, and using EFS where necessary. We don't love it, but the resources to implement something else aren't there now.
Would in a StorageClass go some way to fixing the issue? (At least for new claims anyway)? From Edit: I tried this, and the kube-controller-manager logs say: but the pod is FailedScheduling "2 node(s) didn't find available persistent volumes to bind.".
I solved this in the Elastic Operator by creating StorageClasses per zone and spreading work across that way. You need to make sure your app is zone tolerant. For example, the Elastic operator sets up sharding and replication across zones, given one AZ goes down, the other two still have enough of the data to continue functioning.
doesn't each pod then need access to storage from a different class (appropriate for the zone it is in)? How does that work?
Just ran into this as well, running work load specific zones in 2 regions for development to save cost over 3, had an EBS volume provisioned in the zone with no instance.

should address the above scenarios when you use set in the storageclass. The plumbing for went into EBS in 1.12 as a beta feature.
I have a similar issue: 3 masters + 3 nodes in 3 regions. I have a PV (created on regionA) for data backups created by a CronJob. When the cron's job gets scheduled on a node on a different region (regionA's CPU has been allocated almost fully and the master chooses another node in a different region) - it can't start because the PV is not accessible. Is there a way to avoid this issue?
I am also facing the same issue with 1.10 version of Kubernetes. Nodes are running in two regions, PV got created in the same region. still, I am getting
As mentioned above, this feature is now in beta in 1.12:
Are there any workaround for Amazon EKS to address the above issue? I am trying to make a multi-az JupyterHub deployment and changing zones causes the "no available volume zone" error...
Is this still in progress? I just ran in to issue as well
we moved it to GCP with 1.10 version and didn't face the issue
I'm also having this issue... So there's no solution to this?
im also having this issue on AWS EKS Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m (x841 over 1h) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. losing hope over here :(
Warning FailedScheduling 3m2s (x660 over 58m) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone.
Another similar case here, like 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. Like the use case by but the reason is not memory is pods allocation. The EBS was created in the AZ where the node is full (1c).. you can guess the rest..
Any update on this ? Since Node affinity is a beta feature causing this issue?

Edit: (Beware: Updating your EKS cluster can potentially screw things up & cause nodes and volumes in different zones)
I'm using EKS to deploy my services integrating it with rancher, I ran into the same issue as mentioned above. My setup is something like this: have few labels for the worked nodes to seperate the nodes per namespace/team as each team requires different types of instances. created the nodes for the pods in that node needs persistant volumes in a single availability zone as aws EBS attachment is per availability zone. I deployed my applications, the persistant volume got created but in different availability zone as I used the default storage class which creates EBS in random zones. The issue is fixed by creating another storage class with only single availbility zone.
I am using EKS , and I designed not only my pods to be Cattle, but also my nodes are Cattle as well : I have a Pod created by statefulset. This statefulset defines a PVC. The PVC triggers the creation of new EBS in the same AZ of the Node (i don't have pb here) Because Nodes are Cattle (destoryable any time), it happen that there is no Node exist in the AZ of EBS (PVC) =Pod of statefulset remains in "Pending" state =Pod events shows this error: " 1 node(s) had no available volume zone" Once a New node joins the cluster in the wanted AZ, the Pod of statefulset runs like a charm because the EBS can be attached at that node (same AZ). Thinking to use EFS ( )but looking for better solutions.
from my experience, EFS performance is not that great. better avoid it.
It depends how you use it. It's not great for lots of little files (don't check out a git repo on it, for instance ). It's been working great for us in production for almost a year so far, mainly on larger PDF files etc. But, like anything, definitely test it for your use case.
So I do have a question. If I just put the PVC into the specific availability. zone. Does that allow the other pods on different nodes to use it?

The nodes running on other availability zones couldn't access this PV until if you use EFS rather than EBS
I ran into a similar issue today investigating a error I have a PVC: and originally it bound fine, with a PV in us-east-1d Then, I ran an upgrade and none of the new nodes were created in us-east-1d And, now I have a disk in a zone without any nodes.
I found the issue. The issue is that when you are using AWS then the PVC is set to the Node. That means that if you try to put that pod or deployment on another node then it will not connect with the PVCs and being a pending state. It's an AWS thing. I resolve this by specifying taints and toleration to the nodes within that specific region.
thank you! can you post an example for more elaboration


Question: "fmt" "path/filepath" "strings" <ins> "sync" </ins> "testing" "time"

What happened: Windows kubelet panic occasionally on nodes which have high volume of pods run and delete. Full logs: Preliminary investigate result: How to reproduce it (as minimally and precisely as possible): Run high volume of pods and delete them when they completed in Windows nodes. Environment: Kubernetes version (use ): 1.17.7 Cloud provider or hardware configuration: Azure AKS OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..1282
/assign I would like to give this a shot.
/remove-sig windows I think this is not a windows specific problem. /sig node /area kubelet
Hi! Thanks for your report, I have filed to address this issue.
This issue is a bit urgent for our situation, it occurs every few days in production clusters. Seems the PR is pending a few days, is it possible to increase the priority to merge it trunk and 1.17/1.18 branch? Thanks
I understand how urgent this issue is, but I am not privileged to do anything further :(
Do you know anyone who can help?
looks like a simple fix, add a lock for , I could help.
There is already a fix by , the problem is the PR is pending, not sure if you or someone can increase the priority to merge it trunk and 1.17/1.18 branch asap. Our AKS clusters are affecting by this frequently.
Thanks guys.


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" clientset "k8s.io/client-go/kubernetes"

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: encounteredError = true } } <del> { filterBuf := bytes.NewBuffer(nil) writeLine(filterBuf, "*filter") writeLine(filterBuf, fmt.Sprintf(":%s - [0:0]", kubeServicesChain)) writeLine(filterBuf, fmt.Sprintf("-X %s", kubeServicesChain)) writeLine(filterBuf, "COMMIT") </del> <ins> // Flush and remove all of our chains. iptablesData = bytes.NewBuffer(nil) if err := ipt.SaveInto(utiliptables.TableFilter, iptablesData); err != nil { glog.Errorf("Failed to execute iptables-save for %s: %v", utiliptables.TableFilter, err) encounteredError = true } else { existingFilterChains := utiliptables.GetChainLines(utiliptables.TableFilter, iptablesData.Bytes()) filterChains := bytes.NewBuffer(nil) filterRules := bytes.NewBuffer(nil) writeLine(filterChains, "*filter") for _, chain := range []utiliptables.Chain{kubeServicesChain, kubeForwardChain} { if _, found := existingFilterChains[chain]; found { chainString := string(chain) writeLine(filterChains, existingFilterChains[chain]) writeLine(filterRules, "-X", chainString) } } writeLine(filterRules, "COMMIT") filterLines := append(filterChains.Bytes(), filterRules.Bytes()...) </ins> // Write it. <del> if err := ipt.Restore(utiliptables.TableFilter, filterBuf.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters); err != nil { </del> <ins> if err := ipt.Restore(utiliptables.TableFilter, filterLines, utiliptables.NoFlushTables, utiliptables.RestoreCounters); err != nil { </ins> glog.Errorf("Failed to execute iptables-restore for %s: %v", utiliptables.TableFilter, err) encounteredError = true }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: if got, expected := o.GenerateName, pod.ObjectMeta.GenerateName; got != expected { t.Errorf("expected generatedName to be %q, got %q", expected, got) } <ins> if got, expected := o.Generation, pod.ObjectMeta.Generation; got != expected { t.Errorf("expected generation to be %q, got %q", expected, got) } </ins> if got, expected := o.Labels, pod.ObjectMeta.Labels; !reflect.DeepEqual(got, expected) { t.Errorf("expected labels to be %v, got %v", expected, got) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: import ( "github.com/go-openapi/spec" <ins> "k8s.io/kube-openapi/pkg/aggregator" </ins> ) // MergeSpecs aggregates all OpenAPI specs, reusing the metadata of the first, static spec as the basis. <del> // Later paths and definitions override earlier ones. None of the input is mutated, but input // and output share data structures. func MergeSpecs(staticSpec *spec.Swagger, crdSpecs ...*spec.Swagger) *spec.Swagger { </del> <ins> // The static spec has the highest priority, and its paths and definitions won't get overlapped by // user-defined CRDs. None of the input is mutated, but input and output share data structures. func MergeSpecs(staticSpec *spec.Swagger, crdSpecs ...*spec.Swagger) (*spec.Swagger, error) { </ins> // create shallow copy of staticSpec, but replace paths and definitions because we modify them. specToReturn := *staticSpec if staticSpec.Definitions != nil {

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: Containers: []api.Container{ { Name:  "webserver", <del> Image: "gcr.io/google_containers/nettest:1.4", </del> <ins> Image: "gcr.io/google_containers/nettest:1.3", </ins> Args: []string{ "-service=" + name, //peers >= totalPods should be asserted by the container.

It's failing on both GCE and GKE-CI. It appears to have broken somewhere around build number 2509/2510 which includes a likely candidate PR (). /cc
I was about to debug this one. I'll take it.
Root cause is almost definitely one of these: Summary secrets example pod (details) the permissions on python files. (details) the implementation of DeepCopy to use reflection instead of Gob encode/decode. (details) networking test (details) back to executable. (details) protected docker processes (-900) (details) proxySocket out of to (details) Commit by Fix secrets example pod The file was modified Commit by Fixing the permissions on python files. The file was modified cluster/juju/charms/trusty/kubernetes- The file was modified cluster/juju/return-node- The file was modified cluster/juju/charms/trusty/kubernetes- Commit by Changing the implementation of DeepCopy to use reflection instead of Gob encode/decode. The file was The file was modified The file was modified Commit by Fix networking test The file was modified contrib/for-tests/network- The file was modified contrib/for-tests/network- The file was modified The file was modified contrib/for-tests/network-tester/Makefile Commit by Changing back to executable. The file was modified Commit by OOM protected docker processes (-900) The file was modified The file was modified The file was modified Commit by Move proxySocket out of to Moves proxySocket out of to new in proxy package in order to start separating proxy logic and implementation and make proxier more manageable to review. The file was modified The file was
Pretty sure it's commit , so rolling back now. cc .
The above rollback solved that immediate problem, but now the test, while proceeding healthily, is timing out. More debugging or optimization required: Networking should function for intra-pod communication ..... [irrelevant stuff deleted] ..... INFO: Proxy status call returned in 4.861944ms INFO: Attempt 35/60: test still running INFO: About to make a proxy status call Build timed out (after 90 minutes). Marking the build as aborted. Build timed out (after 90 minutes). Marking the build as failed. Ran 40 of 46 Specs in 5069.444 seconds FAIL!

-- 18 Passed 5 Skipped Build was aborted Recording test results
I'm not entirely sure why, but the test is running normally again, completing consistently in 13 seconds. Closing this issue for now. !


Question: Other details: <del> * [API](api-conventions.md) </del> <ins> * [API](api.md) </ins> * [Client libraries](client-libraries.md) * [Command-line interface](kubectl.md) * [UI](ui.md)

Simple version is done, more complex version needs work. Done, now that readiness corresponds to Running and is tracked by Kubelet status if the readiness hook is set? Done Done Do we want to do a more generic one or a less generic one? OpenShift was looking more at specific topical plugin commands ("add-volume", "set-default-resources"), but not sure how much hand-holding users need. Getting close. Exec has landed and is a replacement for ssh Attach on start remains hard. Implemented
I/we should also file issues for outstanding items without issues, and add lines for newly filed issues (e.g., command flag for run-container). Readiness is not yet used by rollingupdate. With respect to patch on the command line: I was thinking of a generic mechanism, but I could see the case for domain-aware commands, also. I think we need to better factor the command code and library code, and generic functionality vs. domain-aware functionality. Neither should ever convert to the internal representation. Domain-aware functionality (e.g., describe) should target a specific API version. Configuration generators, such as run-container, should be provided as services, or at least as separate plugin commands.
ref
ref
ref
ref
ref
ref


Question: } } <del> func TestFillPodStatus(t *testing.T) { pod := makePod(api.NamespaceDefault, "foo", "machine", "bar") expectedIP := "1.2.3.4" expectedTime, _ := time.Parse("2013-Feb-03", "2013-Feb-03") config := podCacheTestConfig{ kubeletContainerInfo: api.PodStatus{ Phase:  api.PodPending, Host:   "machine", HostIP: "ip of machine", PodIP:  expectedIP, Info: api.PodInfo{ leaky.PodInfraContainerName: { State: api.ContainerState{ Running: &api.ContainerStateRunning{ StartedAt: util.NewTime(expectedTime), }, }, RestartCount: 1, PodIP:        expectedIP, }, }, }, nodes: []api.Node{*makeHealthyNode("machine", "ip of machine")}, pods:  []api.Pod{*pod}, } cache := config.Construct() err := cache.updatePodStatus(&config.pods[0]) if err != nil { t.Fatalf("Unexpected error: %+v", err) } status, err := cache.GetPodStatus(pod.Namespace, pod.Name) if e, a := &config.kubeletContainerInfo, status; !reflect.DeepEqual(e, a) { t.Errorf("Expected: %+v, Got %+v", e, a) } } </del> func TestFillPodInfoNoData(t *testing.T) { pod := makePod(api.NamespaceDefault, "foo", "machine", "bar") expectedIP := ""

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: import ( "context" "fmt" <del> "strings" </del> "time" apps "k8s.io/api/apps/v1beta1"

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details