Question: }) } if mountEtcHostsFile { <del> hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain) </del> <ins> hostAliases := pod.Spec.HostAliases hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain, hostAliases) </ins> if err != nil { return nil, err }

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? Ingress, hosts and /etc/hosts Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request There have already been a few issues regarding modifying : , But both have a different use-case. First some information before I explain my use-case. I have a pod which run a web app (nginx container + php-fpm container) and it available over multiple sub domains. I want request to the exposed sub-/domains (ex. ) to go directly to the local webserver instant of going the long way (Cloudflare -Ingress Controller -Node -Container). There is 2 reason behind this: is faster/reduce overhead. of our app code call the exposed domains, and need to hit the same server, as the request oriented from (the code write to some tmp file, and other PHP code need to read that tmp file, when it get a request. Bad design: properly). cc Kubernetes version (use ): Environment: DigitalOcean, CoreOS.
I'm not sure I understand the use case, especially the 2nd. This is the case of a pod talking to itself via the public hostname? Why not just use localhost?
Public hostname == ingress . Yes, without going through the load balancer, some of the PHP also require "sticky" sessions and then it is just easier if the exposed domains just point to localhost. That require extra logic in the app code and when the app code expose multiple sub-/domains, then it is just easier adding the domain to /etc/hosts.
Do you know all the hostnames at pod startup time, or can they possibly be later? I guess I am not against this idea in principle, I just have a nagging suspicion it will blow in all the corner cases. I guess we could look at a community implementation, if someone did one. I don't think that we have time on our side right now to drive it.
I currently have a script like this: , which "extract" the hostnames from the nginx config at container startup (and "patch" the location of the file). It also configure to forward some request internally (we don't expose all domains externally).

-- I'm not sure about your particular use case as the host name extraction logic seems to be quite complex (parsing of the nginx config). The two things I do see that are useful: Ability for a ConfigMap to be written out as a single file into the container instead of injecting it as a directory of files. This would solve a lot of problems where a piece of the container configuration needs to be trivially changed. I'm fairly certain this is already filed, I can't seem to find the issue. Inheritance of host entries from the outer node.
I think the use-case beyond service is if the hosts need to be loopback or needs to be configured per pod. I tried to understand how is controlled by kubelet, and it seems like this feature would not be hard to implement. I'm willing to spend some time on this. in terms of implementation: from my reading, host mount is done by inside , and the content of the hosts file is controlled by the methods that call them all have access to the pod object. it seems like we can add a field to PodSpec (can be done as an Pod-level alpha annotation to start): or if we want something more explicit: that can be read into and , which then mounts as to the Pod.
Let me know if ^ makes sense. If so, I can make a formal proposal in or just a PR, whichever is appropriate.
I don't think this warrants a proposal or feature. I would argue for the more expliciti form of the API, but hostnames has to be plural, and IP has to be the mergeKey. or - any objections? Consider from a CRI point of view?
For me, this solves my problems.
so the struct would be something like where the keys are IP addresses, and values are hostnames. I presume this field would live under PodSpec. is the key we want to surface? ? the content would be: which after some merging would look like: (the go struct represented in yaml) and the output could either look like: or I think the latter is better since it's one line per entry, and seems to follow existing convention in .
Your proposal can definitely work for my use-case, although I would love if I could configure it from a init container, but this could work.

Keep up the good work :)
I wouldn't try to massage it into a map, just: , Kristian Klausen
So you'd be passing in into . That makes sense. This might be a stretch, but what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP.
This is currently handled completely by kubelet, so it wouldn't be an issue for CRI. kubelet doesn't mount if the pod is using the host network, so this would either need to be changed, or validation should reject the pod if this is not allowed. If you need to dynamically change the file after the container starts, I think suggestion of trying to make ConfigMap work would be better.
I think validation to reject the option if the pod is using the host network makes sense. is still a static file though. And if you inject a file that replaces , what are you going to do with existing dynamic hostnames that are by kubelet?
I don't have any concern from CRI perspective, and the proposal look fine to me from the high level. what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP. But can I request not stretching the feature to support dynamic changes? Pod can be deleted, and re-created, thus the pod's IP might be changed. Currently we only allow container image being updated in PodSpec. Unless we allow updating the new field, let's call it etcHost for now, in PodSpec, it won't work. Even we allow that field being updated, I think that is overkill.
With regards to the dynamicness, I'm just trying to design the API in such a way that's flexible. The field in PodSpec won't be dynamic. I imagine something like this: (The idea is borrowed from ) validation ensures that is only present if type is . then IP is gathered at Pod creation time. In kubelet's :
Why do you need pod IP? Why not use 127.0.0.1 ?

, Ricky Pai <:
The use-case that i'm more interested in is support for nodeIP, but I'm using podIP as an example because it's easier to illustrate with current code. But yes I agree with you that pod IP might not make sense since you have access to localhost already. I can imagine nodeIP being used to talk to daemons that are shared across pods. is a proper fix for the use-case, but this could be one way to address it.
sorry for the scope-creep. I'm just afraid using IP as a string field makes it hard to add support for other IP sources should that ever be an option. but it's probably premature right now. for now, I'll just proceed with:
Given that we have a way to get node IP through downward API, why do we need that here? On Apr 13, 2017 6:09 PM, "Ricky Pai" <:
Ideally, if the applications running inside the containers have flexible ways to configure dependent hostnames like through environment variables, then downward API would be sufficient. We can probably extend the argument that the whole support for is moot if applications supported flexible injection of dependent hostnames. The problem is many applications don't provide this level of flexibility. Hostnames are often hard-coded inside config files and the client libraries. Outside of Kubernetes, I might use a script to manage and expect nothing else to change the file; while it is possible that I use an init container to do the same modification to , since the mounting of happens whenever a container is restarted, I risk the file being changed and my application not being able to connect to its dependent services. This creates much headache migrating applications from a more lenient environment to Kubernetes, which is what we've been facing. I think support is the right direction and is why I'm willing to commit time to make it happen.
We have another issue open for "node-local Services" which sounds like what you want. I am not a big fan of the expansion of host mappings like this. I'd rather go all the way and give nodes a static link-local address or else solve node-local services. , Ricky Pai <:
Thanks for all the feedback I'm well aware of node-local services, but it seems like progress on the proposal is stalled.

I believe node-local services would be the proper way to address the use case I described, especially with the "locality" concept you described in that can be expanded beyond the node. However, in lieu of node-local services, with references to the node IP might be a quick way to address the use case. Other tickets describing the need for node-level sidecars also proposed either using node-local services or : I also understand this complicates the interface a little bit more, but it would really benefit cluster operators. If you still think this is inappropriate, then I can proceed with this interface next week:
I don't have a strong opinion on this topic but it seems that it should be addressed as part of what is the status of the proposal?
I am not familiar with the node-local services and its progress, but if this can satisfy the use cases, I'd suggest that not adding new fields to the pod API for a quick workaround.
I feel the original proposal got digressed by my zealous talk of node IP support. Allow me to clarify: serves several use cases: external hostname with a specific IP. i.e. if I want to say mock out an external hostname like a hostname with localhost. i.e. if I want to override to point to we allow dynamic injection of nodeIPs, it can be an easy way for pods to talk to nodes in absence of node-local services. only the third use-case is relevant to node-local services. I think it's still useful to support adding entries to even with static IPs
I understand the third case, but I'd like to consider other options to solve that, first. I think the rest is uncontentious. , Ricky Pai <:
Thanks. I'll get the changes up this week. There'll likely be three changes: tests for existing hosts file content (there's no tests currently) change for on to use in Is there any possibility to cherry-pick those back in 1.6? Or would this have to wait for 1.7?
AFAIK, we don't cherrypick features.
has been created for the API change
just to clarify, are you in favor of adding HostMappings to the API?
I am neutral on it. It's come up often enough that i believe the problem is real. The API doesn't seem harmful to me, what do you think?

You're the guardian of CRI... , Yu-Ju Hong <:
Also the name is terrible - help? HostAliases? DNSAliases? , Tim Hockin <: solve that, first. I think the rest is uncontentious. , just to clarify, are you in favor of adding HostMappings to the API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I'm up for . might accidentally imply that it's actually interfacing with kube-dns.
I am neutral but just wanted to make sure there are no obvious, alternative ways to achieve this. BTW, the proposed solution changes the k8s API, but not CRI :) On the other hand, if we support this on the k8s API level, we should discourage users from modifying this file directly. For example, reported a bug where users modify in the init containers, and later found that kubelet overwrites it.
We should definitely discourage modifying that, but we should perhaps respect it if a user provided a volume mount. Thoughts on name? , Yu-Ju Hong <:
I am terrible at naming... I agree that having DNS in the name can be misleading. sounds ok. might be more direct?
I'd go for brevity when possible. HostAliases ? On Apr 27, 2017 5:27 PM, "Yu-Ju Hong" <:
I'm updating the PR to use . thanks for the suggestion
Just posted the last PR to enable this feature:
could you add a node e2e test for this feature?
Do we need e2e? Every e2e is a liability, and slows down test runs - e2e is needed if we can't cover in unit or integration. , Yu-Ju Hong <:
Unit tests are great to ensure the individual functions work correctly, but so far it's not sufficient to prevent regression in our codebase from my experiences. Since this is a new addition/feature to the API, I'd prefer having a corresponding node e2e test. Integration test can be sufficient, but I've not had any pleasant experience debugging integration tests. Maybe that has changed already?
no, integrtaion tests are under-serviced tech-wise.

, Yu-Ju Hong <:
does a simple node e2e test sound good to you?
SGTM , Yu-Ju Hong <:
I saw that the HostAlias's comment: I'm wondering why it couldn't work to hostNetwork pods? And how does a hostNework pod initialize its /etc/hosts file?
Are we still missing a test for this? I'd hate to have to kick it out of the 1.7 release for lack of tests..
yes we are still missing a test. what's the deadline for 1.7 release? sorry i was on PTO and dropped the ball
hostNetwork pods uses the host's file, while only non-hostNetwork pods have managed by kubelet. because kubelet does not manage the file when using hostNetwork, kubelet can't write additional entries from hostAliases.
Just chatted on slack, for containers using the host network, docker actually makes a copy of "/etc/hosts" on the host for each of them. I think kubelet can potentially support HostAliases for hostnetwork pods by copying and mounting the file. Not sure if there are be other concerns though.
if you make the change to make kublet copy and manage for hostNetwork pods, I can make the change for HostAliases. one concern: currently, the logic includes , including from and . Should this be allowed? if not, there should probably be validation disallowing those field when hostNetwork is used. there currently isn't any unless I read the code wrong.
node e2e test:
THANKS! , Ricky Pai <:
That would be a great feature! Currently we have pods using hostNetwork and using init containers to restore some host mappings -- basically . With HostAliases feature it would be fantastic fit!
We are using and for hostNetwork pods.
This is something I don't understand in the code: It seems that it is still mounting "/etc/hosts" for hostnetwork pods if . Does that mean kubelet is still managing it?
One of our current assumptions is that kubelet will mount a shared /etc/hosts (which is not node's) for all containers within a pod in hostNetwork mode. Which seems to be the codepath in the code linked above. Is this holding true?
I checked the git history and that was just an artifact caused by switching from internal to v1 API objects.

The original code is: In the internal API, is a field in , so you can see why had to be non-nil. In the v1 API, is a field in the pod spec: . When switching all code to use the versioned (v1) objects, this got lost in translation. I believe the apiserver sets the default pod SecurityContext, so it should never be nil. I'll send a PR to remove the redundant check.
Thanks for looking into that. Currently even with hostNetwork pods, it will have mounts for . Checking within the pod: If the check is removed, what would happen on the shared mount of /etc/hosts ? My concern and question is about the guarantee here. Is there still a copy of which is shared and mounted for all containers within a pod?
Removing the check itself won't affect anything since securitycontext should always be non-nil. That mount for hostnetowrk pods was by docker (not by kubelet) so it should be one per container, and not shared by containers within a pod. I didn't try it myself, but it should be easy to verify.
Tried and verified that was shared between containers within a pod for k8s 1.6.4 .
Also verified that modifying for a hostNetwork pod did not change the file on the node.
Then docker did the right thing and use the same file for containers sharing the same network namespace :-)
That makes sense.
Will HostAlias support for HostNetwork pod happen in 1.7 ?
I'm trying to get in so it can make the 1.7 release deadline. help reviewing the PR would be great.
in a multi-container hostNetwork pod, what happens when you change the file, then restart one of the containers? in non-hostNetwork pod, it would revert the changes to
anyways, i'm also in favor of kubelet managing the file even for hostNetwork pods. that way, we can use features like hostAliases, and also then we don't depend on the container runtime to dictate the behavior of the hosts file. it's not clear whether running rktnetes with hostNetwork pod would yield the same result
Is this a dup of ? How are they different?
HostAlias is useful to provide local overrides for a particular Pod, but it seems like would want the alternative DNS name of a service accessible for all Pods.

re: documentation, i'm thinking either adding notes about HostAliases in or as a separate page under "Services, Load Balancing, and Networking"
Is there any issue created for supporting HostAlias in hostnetwork mode?
I can write up what I think will be needed to support HostAlias in hostNetwork pods
This is now in 1.7:
filed issues to follow up with HostNetwork support:
Thank you for this feature. Our use case is different. On an nginx/php-fpm set up where our local dev is via docker-compose yet we're deploying into kubernetes, we can use the exact same nginx config on both environments. We're currently working around this via HOSTALIASES (alias or or whatever the php container is into , which satisfies both deployment types), but this is hacky and doesn't work in Alpine.
the discrepancy between dev/prod would probably cause problems beyond just difference in networking setup. have you considered using to have parity between the two environments?
We have, and found it way too cumbersome for local development. It's way heavier in resources as well - obviously there's a lot more than just docker needed for minikube to run, and most of our developers use Mac, where all this is virtualised. In contrast, docker-compose is a lot easier, more concise, and we have yet to find any environment-y issues caused by difference between these two, except for this one issue we fortunately have a workaround for (HOSTALIASES). We use the exact same base container images for development as well as builds. From the point of view of the services themselves, there's no difference at all being run on one environment or the other - they talk to each other via HTTP or queues.
that's unfortunate. I guess minikube does have overhead since you have to run it inside a VM on your mac machines. But since all components are written in go and cross-compiled in darwin, I wonder if you could setup your own etcd (homebrew has this), then your own kubelet. Docker you can run Docker on Mac, then just have kubelet talk to the local docker socket. This sounds like it would work with lower resource overhead than minikube since you don't to run in a VM. Either way, I'm glad HostAlias is able to address your problem.

I just wish you didn't have to deal with this discrepancy in the first place.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: <del> #!/bin/bash # Copyright 2014 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Generates secret, creates secret on kube, creates pod on kube set -o errexit set -o nounset set -o pipefail ORIGIN=$(dirname "${BASH_SOURCE}") KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. ## Generate resources ${ORIGIN}/resource-generator.sh ## Create the secret ${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/secret.json ## Create the pod ${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/pod.json ## Create the services ${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/api-service.json ${KUBE_ROOT}/cluster/kubectl.sh create -f ${ORIGIN}/ui-service.json </del>  No newline at end of file

This PR was prematurely merged: Need to update the PR based on the actual steps that were needed to get what was demonstrated at the public Kubernetes meet-up last week.
Any chance you can get rid of the hostPort from the example?
- depending on the Kubernetes provider, yes. For Vagrant based provider, I think I still need the hostPort because I do not have routes. For the GCE provider, I should be able to get rid of it because I have routes. In both providers, there were a number of setup steps that did not make the merged PR. For those that are tracking this issue (which I know folks have pinged me on separately) needs to be configured to allow privelaged containers needs to be configured to allow pushing to an insecure registry (this is point in time) Ideally, I could use a ReplicationController in the example to load balance OpenShift, but until we can provide a pattern to run etcd as a pod, I am unable to do this and need to use the embedded etcd we end up running.
Note: I will resolve this issue in 2 phases. Vagrant (which was demonstrated) to document using GCE (which has some additional steps on firewall rules)
Turns out OpenShift Origin is too secure now ;-) This is blocked until OpenShift Origin allows you to mint certs on the client, and then supply the certs on startup as a secret to the pod. This feature is coming soon. /cc
I now have all the required pieces to get this to work. I hope to get a PR submitted to fix this issue by end of this week.


Question: ExpectedThreeWay:       `{}`, ExpectedThreeWayResult: `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, }, <ins> "no diff even if modified null": { Original: `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Current:  `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Modified: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{"key":null},"name":"foo","scalar":true}`, ExpectedTwoWay:         `{"complex_nullable":{"key":null}}`, ExpectedTwoWayResult:   `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{},"name":"foo","scalar":true}`, ExpectedThreeWay:       `{"complex_nullable":{"key":null}}`, ExpectedThreeWayResult: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{},"name":"foo","scalar":true}`, }, "discard nulls in nested and adds not nulls": { Original: `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Current:  `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Modified: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{"key":{"keynotnull":"value","keynull":null}},"name":"foo","scalar":true}`, ExpectedTwoWay:         `{"complex_nullable":{"key":{"keynotnull":"value","keynull":null}}}`, ExpectedTwoWayResult:   `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{"key":{"keynotnull":"value"}},"name":"foo","scalar":true}`, ExpectedThreeWay:       `{"complex_nullable":{"key":{"keynotnull":"value","keynull":null}}}`, ExpectedThreeWayResult: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":{"key":{"keynotnull":"value"}},"name":"foo","scalar":true}`, }, "discard if modified all nulls": { Original: `{}`, Current:  `{}`, Modified: `{"complex":{"nested":null}}`, ExpectedTwoWay:         `{"complex":{"nested":null}}`, ExpectedTwoWayResult:   `{"complex":{}}`, ExpectedThreeWay:       `{"complex":{"nested":null}}`, ExpectedThreeWayResult: `{"complex":{}}`, }, "add only not nulls": { Original: `{}`, Current:  `{}`, Modified: `{"complex":{"nested":null,"nested2":"foo"}}`, ExpectedTwoWay:         `{"complex":{"nested":null,"nested2":"foo"}}`, ExpectedTwoWayResult:   `{"complex":{"nested2":"foo"}}`, ExpectedThreeWay:       `{"complex":{"nested":null,"nested2":"foo"}}`, ExpectedThreeWayResult: `{"complex":{"nested2":"foo"}}`, }, "null values in original are preserved": { Original: `{"thing":null}`, Current:  `{"thing":null}`, Modified: `{"nested":{"value":5},"thing":null}`, ExpectedTwoWay:         `{"nested":{"value":5}}`, ExpectedTwoWayResult:   `{"nested":{"value":5},"thing":null}`, ExpectedThreeWay:       `{"nested":{"value":5}}`, ExpectedThreeWayResult: `{"nested":{"value":5},"thing":null}`, }, "nested null values in original are preserved": { Original: `{"complex":{"key":null},"thing":null}`, Current:  `{"complex":{"key":null},"thing":null}`, Modified: `{"complex":{"key":null},"nested":{"value":5},"thing":null}`, ExpectedTwoWay:         `{"nested":{"value":5}}`, ExpectedTwoWayResult:   `{"complex":{"key":null},"nested":{"value":5},"thing":null}`, ExpectedThreeWay:       `{"nested":{"value":5}}`, ExpectedThreeWayResult: `{"complex":{"key":null},"nested":{"value":5},"thing":null}`, }, "add empty slices": { Original: `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Current:  `{"array":[1,2,3],"complex":{"nested":true},"name":"foo","scalar":true}`, Modified: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":[],"name":"foo","scalar":true}`, ExpectedTwoWay:         `{"complex_nullable":[]}`, ExpectedTwoWayResult:   `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":[],"name":"foo","scalar":true}`, ExpectedThreeWay:       `{"complex_nullable":[]}`, ExpectedThreeWayResult: `{"array":[1,2,3],"complex":{"nested":true},"complex_nullable":[],"name":"foo","scalar":true}`, }, "filter nulls from nested slices": { Original: `{}`, Current:  `{}`, Modified: `{"complex_nullable":[{"inner_one":{"key_one":"foo","key_two":null}}]}`, ExpectedTwoWay:         `{"complex_nullable":[{"inner_one":{"key_one":"foo","key_two":null}}]}`, ExpectedTwoWayResult:   `{"complex_nullable":[{"inner_one":{"key_one":"foo"}}]}`, ExpectedThreeWay:       `{"complex_nullable":[{"inner_one":{"key_one":"foo","key_two":null}}]}`, ExpectedThreeWayResult: `{"complex_nullable":[{"inner_one":{"key_one":"foo"}}]}`, }, "filter if slice is all empty": { Original: `{}`, Current:  `{}`, Modified: `{"complex_nullable":[{"inner_one":{"key_one":null,"key_two":null}}]}`, ExpectedTwoWay:         `{"complex_nullable":[{"inner_one":{"key_one":null,"key_two":null}}]}`, ExpectedTwoWayResult:   `{"complex_nullable":[{"inner_one":{}}]}`, ExpectedThreeWay:       `{"complex_nullable":[{"inner_one":{"key_one":null,"key_two":null}}]}`, ExpectedThreeWayResult: `{"complex_nullable":[{"inner_one":{}}]}`, }, "not filter nulls from non-associative slice": { Original: `{}`, Current:  `{}`, Modified: `{"complex_nullable":["key1",null,"key2"]}`, ExpectedTwoWay:         `{"complex_nullable":["key1",null,"key2"]}`, ExpectedTwoWayResult:   `{"complex_nullable":["key1",null,"key2"]}`, ExpectedThreeWay:       `{"complex_nullable":["key1",null,"key2"]}`, ExpectedThreeWayResult: `{"complex_nullable":["key1",null,"key2"]}`, }, </ins> "added only": { Original: `{"name":"foo"}`, Current:  `{"name":"foo"}`,

Note: I'm using OpenShift. s/oc/kubectl/ below if that's how you roll. I'm using a command similar to the following to idempotently ensure a label is absent on a Namespace object: If the object's contains anything other than , this works fine: the resulting object has no key. However, if is initially absent, this command adds it with an entry with the empty string as a value: Running the command again removes the entry and the . Using this logic in a controller causes a hot loop as we thrash back and forth between the above and no labels. Using a value in a dict in a StrategicMerge patch should idempotently ensure the key is absent, even if the dict is initially absent/empty. I didn't try this on anything other than a Namespace; and I didn't try it on any dict other than . Workaround: Add any other label, as seen above. <details</details<detailsAWS </details<details</details<details</detailsI believe the issue is on the server, but... v0.9.3-0.- </details<details</details
/sig api-machinery
/assign /cc /triage accepted
can you look into it?
It looks that this idempotency problem is fixed in latest releases(after being label as you mentioned). All responses are legit; Kubernetes Version
The bug happens when there are no other labels. Are you saying the "fix" was for kubernetes to always add a label to the object?
That was an incidental change specific to namespaces. I'd expect the issue still exists for other types
you are both right. This is incidental "fix" for namespaces. I'll look at others to find the actual problem.


Question: for portname := range portsToEndpoints { svcPort := proxy.ServicePortName{NamespacedName: types.NamespacedName{Namespace: endpoints.Namespace, Name: endpoints.Name}, Port: portname} <del> glog.V(2).Infof("LoadBalancerRR: Removing endpoints for %s", svcPort) // If the service is still around, reset but don't delete. if state, ok := lb.services[svcPort]; ok { state.endpoints = []string{} state.index = 0 state.affinity.affinityMap = map[string]*affinityState{} } </del> <ins> lb.resetService(svcPort) </ins> } }

/kind bug What happened: We constantly log message like at V(2) that don't tell any useful information.


Question: } } <del> func (h *hostpathSymlinkDriver) GetDriverInfo() *DriverInfo { </del> <ins> func (h *hostPathSymlinkDriver) GetDriverInfo() *DriverInfo { </ins> return &h.driverInfo } <del> func (h *hostpathSymlinkDriver) SkipUnsupportedTest(pattern testpatterns.TestPattern) { </del> <ins> func (h *hostPathSymlinkDriver) SkipUnsupportedTest(pattern testpatterns.TestPattern) { </ins> } <del> func (h *hostpathSymlinkDriver) GetVolumeSource(readOnly bool, fsType string) *v1.VolumeSource { // hostpath doesn't support readOnly volume </del> <ins> func (h *hostPathSymlinkDriver) GetVolumeSource(readOnly bool, fsType string) *v1.VolumeSource { // hostPathSymlink doesn't support readOnly volume </ins> if readOnly { return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: configs[NonRoot] = Config{promoterE2eRegistry, "nonroot", "1.1"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go configs[Pause] = Config{gcRegistry, "pause", "3.4.1"} <del> configs[Perl] = Config{dockerLibraryRegistry, "perl", "5.26"} </del> <ins> configs[Perl] = Config{promoterE2eRegistry, "perl", "5.26"} </ins> configs[PrometheusDummyExporter] = Config{gcRegistry, "prometheus-dummy-exporter", "v0.1.0"} configs[PrometheusToSd] = Config{gcRegistry, "prometheus-to-sd", "v0.5.0"} configs[Redis] = Config{promoterE2eRegistry, "redis", "5.0.5-alpine"}

Part of umbrella to migrate kubernetes e2e test images/registries to community-owned infrastructure: The following images are used by 1.20 e2e tests We should migrate them to community owned infrastructure (ref: for deciding how to implement a mirror registry) We should consider doing this for release branches as well, which may use different images/tags. So these wouldn't necessarily be cherry-picks Two possible approaches: change the registry in source (I would prefer eliminating the extra "docker library" registry, and consolidate to a single e2e test image registry) start running tests with a config that swaps out for some other registry /sig testing /sig release /area release-eng /wg k8s-infra /milestone v1.21
/kind cleanup
/triage accepted /help cc: Give a shout if any of you are interested in working on this!
I'm in! /assign
/remove-help
cc
/cc
I've had 2 PRs opened for this, before it was decited that the test jobs would use the registry mirror configuration. Has there been any consensus on the approach? If so, I can un-abandon these PRs if necessary:
ooh thanks for sharing I don't think there has been any consensus, though I personally think your work on kubernetes/kubernetes is relevant, especially that you've covered everything except gluster and redis.
We already have the redis image here [1], we'll just have to trigger a job for it afterwards. Indeed, I was missing the gluster image, I only considered the image for it.
so I've been thinking about this for a bit and I'm a bit concerned with the visibility of this setup to test authors -- I'm not sure if this is straightforward / intuitive enough for them to follow. alternatively, are there other lighter (in terms of maintenance) solutions that we can pursue?
Hm, what do you mean by the visibility of this setup of test authors? If I'm not mistaken, the current goal is to not rely so heavily on dockerhub for testing anymore. For this purpose we are mirroring these images into our own registries, and then we are going to update to use our own registry instead of the dockerhub one. Test authors shouldn't be impacted by this, IMO. From their point of view, it's the same image, from a different source.

And in terms of maintenance... hm, techically, if someone would want to use an newer image, say , so it would be consistent.
yes, I think we're on the same page on this yeah what I'm wondering is: are there ways we make it even less significant work on the first step of modifying and files? if that step could disappear altogether, especially in scenario of adding new images, that would be great.
I don't really see how. Other images are doing something similar too. For example, the pause image also has its BASEIMAGEs and VERSION in its Makefile. Not all images are built exactly the same, or have the same base image, or are built for the same arch types. Some of them are from scratch, others are from an alpine image, others are from a different alpine image, others are from a debian image... and then we also have windows base images. So, having those BASEIMAGE files addresses this issue. Plus, it also allows us to control what platforms we build for and what we test for. For example, the s390x support wasn't there from the start, it was afterwards. Another example, we want to add the base images at some point. Anyways, modifying the BASEIMAGE files is not very common, but when it is, there's a good reason behind it, so it can't really be optional. Plus, adding / modifying the BASEIMAGE and VERSION files is not significant work. For example, mirroring the 7 images in this PR [1] probably took me ~10 minutes or so. From what I can tell, you want some fast way to just say: "mirror me this image entirely onto our registry". I don't know if we have something like that, and if we do, it'll probably also mirror images with arch types we don't need, and most likely won't have images for platforms we actually need (for example, the ` images on dockerhub does not have any Windows images, but we need them). The test images building process for the test images was not designed for mirroring images, but it can easily do it and we can use it to do so. In addition to that, we can even extend those manifest lists with Windows support [2]. [1] [2]
oh right, apologies for the confusion.

I was referring to having a Dockerfile as a whole, but your explanation is very helpful for me to gain more context about image usage in our infrastructure -- thank you for elaborating I agree the what isn't significant, but the where worries me a bit. I guess in some ways I kind of wish that there is way for this work to simply exist in without touching k/k proper, though now I'm not too sure if that would be overall less complex than this approach. yeah, I was trying to find more solutions towards this direction though I haven't found anything significant which aligns with this topic.
/priority important-soon
In order to complete this, we are going to need these 2 trivial PRs: Retrigger image builder jobs for some images: Small fix for Windows httpd images:
seems like much of the work has been done by Claudiu or I do not have access to finish -- I'll unassign myself to keep this aligned with the state of things /unassign
/assign who has done a lot of the work for this


Question: <ins> // +build integration,!no-etcd /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package integration // This file tests the scheduler. import ( "net/http" "net/http/httptest" "testing" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" "github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/client/record" "github.com/GoogleCloudPlatform/kubernetes/pkg/master" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/admission/admit" "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/scheduler" _ "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/scheduler/algorithmprovider" "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/scheduler/factory" ) func init() { requireEtcd() } func TestUnschedulableNodes(t *testing.T) { helper, err := master.NewEtcdHelper(newEtcdClient(), testapi.Version()) if err != nil { t.Fatalf("Couldn't create etcd helper: %v", err) } deleteAllEtcdKeys() var m *master.Master s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) defer s.Close() m = master.New(&master.Config{ EtcdHelper:        helper, KubeletClient:     client.FakeKubeletClient{}, EnableLogsSupport: false, EnableUISupport:   false, EnableIndex:       true, APIPrefix:         "/api", Authorizer:        apiserver.NewAlwaysAllowAuthorizer(), AdmissionControl:  admit.NewAlwaysAdmit(), }) client := client.NewOrDie(&client.Config{Host: s.URL, Version: testapi.Version()}) schedulerConfigFactory := factory.NewConfigFactory(client) schedulerConfig, err := schedulerConfigFactory.Create() if err != nil { t.Fatalf("Couldn't create scheduler config: %v", err) } eventBroadcaster := record.NewBroadcaster() schedulerConfig.Recorder = eventBroadcaster.NewRecorder(api.EventSource{Component: "scheduler"}) eventBroadcaster.StartRecordingToSink(client.Events("")) scheduler.New(schedulerConfig).Run() defer close(schedulerConfig.StopEverything) DoTestUnschedulableNodes(t, client) } func podScheduled(c *client.Client, podNamespace, podName string) wait.ConditionFunc { return func() (bool, error) { pod, err := c.Pods(podNamespace).Get(podName) if errors.IsNotFound(err) { return false, nil } if err != nil { // This could be a connection error so we want to retry. return false, nil } if pod.Spec.Host == "" { return false, nil } return true, nil } } func DoTestUnschedulableNodes(t *testing.T, client *client.Client) { node := &api.Node{ ObjectMeta: api.ObjectMeta{Name: "node"}, Spec:       api.NodeSpec{Unschedulable: true}, } if _, err := client.Nodes().Create(node); err != nil { t.Fatalf("Failed to create node: %v", err) } pod := &api.Pod{ ObjectMeta: api.ObjectMeta{Name: "my-pod"}, Spec: api.PodSpec{ Containers: []api.Container{{Name: "container", Image: "kubernetes/pause:go"}}, }, } myPod, err := client.Pods(api.NamespaceDefault).Create(pod) if err != nil { t.Fatalf("Failed to create pod: %v", err) } // There are no schedulable nodes - the pod shouldn't be scheduled. err = wait.Poll(time.Second, time.Second*10, podScheduled(client, myPod.Namespace, myPod.Name)) if err == nil { t.Errorf("Pod scheduled successfully on unschedulable nodes") } if err != wait.ErrWaitTimeout { t.Errorf("Failed while waiting for scheduled pod: %v", err) } // Make the node schedulable and wait until the pod is scheduled. newNode, err := client.Nodes().Get(node.Name) if err != nil { t.Fatalf("Failed to get node: %v", err) } newNode.Spec.Unschedulable = false if _, err = client.Nodes().Update(newNode); err != nil { t.Fatalf("Failed to update node: %v", err) } err = wait.Poll(time.Second, time.Second*10, podScheduled(client, myPod.Namespace, myPod.Name)) if err != nil { t.Errorf("Failed to schedule a pod: %v", err) } err = client.Pods(api.NamespaceDefault).Delete(myPod.Name) if err != nil { t.Errorf("Failed to delete pod: %v", err) } } </ins>

This was broken by and then (hopefully) fixed by . But we need a test for it. cc


Question: return resp, nil } <del> doc, err := html.Parse(reader) </del> <ins> urlRewriter := func(targetUrl string) string { return t.rewriteURL(targetUrl, req.URL) } err := rewriteHTML(reader, writer, urlRewriter) </ins> if err != nil { <del> glog.Errorf("Parse failed: %v", err) </del> <ins> glog.Errorf("Failed to rewrite URLs: %v", err) </ins> return resp, err } <del> t.scan(doc, func(n *html.Node) { t.updateURLs(n, req.URL) }) if err := html.Render(writer, doc); err != nil { glog.Errorf("Failed to render: %v", err) } </del> resp.Body = ioutil.NopCloser(newContent) // Update header node with new content-length // TODO: Remove any hash/signature headers here?

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: return cmd } <del> func runDeleteContext(out io.Writer, configAccess clientcmd.ConfigAccess, cmd *cobra.Command) error { </del> <ins> func runDeleteContext(out, errOut io.Writer, configAccess clientcmd.ConfigAccess, cmd *cobra.Command) error { </ins> config, err := configAccess.GetStartingConfig() if err != nil { return err

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, this is a bug report. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: MacBook Pro Mid-2015 OS (e.g. from /etc/os-release): macOS Kernel (e.g. ): Install tools: Others: N/A What happened: When deleting the context that was my current context, I entered a state where I am seemingly unable to set my context. The shows none as the current: Also, when trying to take any actions that require a context the following error is presented: When viewing the full config () I can see that my selected context is still the one I deleted. What you expected to happen: I believe there are two expectations I had: should allow me to delete my current context, but not enter an inconsistent state in the process -or- should either provide a warning or disallow removing your current context How to reproduce it (as minimally and precisely as possible): I believe that you just need a configuration with multiple contexts. I had three before deleting the context that has me unable to set a new context.

list all contexts to find your current context () delete your current context list your current contexts again, and you'll see no contexts are created try to set your context with one of the ones that exist () listing your contexts again will show none being set Anything else we need to know: 3 contexts were configured before starting to delete one 2 of them were GKE clusters, automatically set up by the binary 1 is a context set up by the context I deleted was one of the contexts The issue was resolved by manually setting my context by editing .
I think this is the confusion of doesn't switch contexts, it lets you set fields on a context. is what switches contexts.
It's technically valid to have no current context, but I agree a warning message like "warning: this removed your active context, use to select a different one" would go a long way.
SGTM.


Question: } // evictionMessage constructs a useful message about why an eviction occurred, and annotations to provide metadata about the eviction <del> func evictionMessage(resourceToReclaim v1.ResourceName, pod *v1.Pod, stats statsFunc) (message string, annotations map[string]string) { </del> <ins> func evictionMessage(resourceToReclaim v1.ResourceName, pod *v1.Pod, stats statsFunc, thresholds []evictionapi.Threshold, observations signalObservations) (message string, annotations map[string]string) { </ins> annotations = make(map[string]string) message = fmt.Sprintf(nodeLowMessageFmt, resourceToReclaim) <ins> quantity, available := getThresholdMetInfo(resourceToReclaim, thresholds, observations) if quantity != nil && available != nil { message += fmt.Sprintf(thresholdMetMessageFmt, quantity, available) } </ins> containers := []string{} containerUsage := []string{} podStats, ok := stats(pod)

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra on 6-Nov-2015 </del> <ins> ###### Auto generated by spf13/cobra on 10-Nov-2015 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_run.md?pixel)]()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16


Question: clientv1core "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/record" "k8s.io/klog" <ins> "k8s.io/kubernetes/pkg/api/legacyscheme" </ins> "k8s.io/kubernetes/pkg/scheduler" schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" "k8s.io/kubernetes/pkg/scheduler/factory"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: I0116 21:50:32. 3959 ] GET /healthz: (1.0472ms) 500 goroutine 2352 [running]: (respLogger).recordStatus(0xc001fa7110, 0x1f4) +0xd2 (respLogger).WriteHeader(0xc001fa7110, 0x1f4) +0x35 (baseTimeoutWriter).WriteHeader(0xc00382ad80, 0x1f4) +0xaf net/http.Error(0xdc0d5b8, 0xc0036a21c0, 0xc0003b5a40, 0x136, 0x1f4) +0xda (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x3aa net/http.HandlerFunc.ServeHTTP(0xc0027a40e0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (pathHandler).ServeHTTP(0xc00278a900, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x510 (PathRecorderMux).ServeHTTP(0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x8a (0x4c243b1, 0xe, 0xc0008aa360, 0xc00014f340, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x661 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x4c3 net/http.HandlerFunc.ServeHTTP(0xc0000da080, 0xdc0d5b8,

0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x459 net/http.HandlerFunc.ServeHTTP(0xc00071c9c0, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x1eeb net/http.HandlerFunc.ServeHTTP(0xc0000da700, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ef00) +0x44 (0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00) +0x476 net/http.HandlerFunc.ServeHTTP(0xc0008a9400, 0xdc0d5b8, 0xc0036a21c0, 0xc00380ee00) +0x44 (timeoutHandler).ServeHTTP.func1(0xc003637560, 0xc00045dd00, 0x6b6d240, 0xc0036a21c0, 0xc00380ee00) +0xb3 created by (timeoutHandler).ServeHTTP +0x1b0 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x25f7a93] goroutine 3306 [running]: (Scheduler).bind(0xc00002db60, 0xc004ae3c00, 0xc0049d0a80, 0x0, 0x1c4efe1) +0x643 (Scheduler).scheduleOne.func1(0xc001ffe701, 0xc00002db60, 0xc004ae3c00, 0x6b51d40, 0xc003a1c1c0, 0xc001ffe700, 0x11, 0x64, 0x64,

0xbf07eb8a4781e3e8, .) +0x21f created by (Scheduler).scheduleOne +0xb6f exit status 2 FAIL 4.409s !!! Error in ./test- Error in ./test- 'go test - -test.timeout=60m -' exited with status 1 Call stack: 1: ./test- main(...) Exiting with status 1 Anything else we need to know: run shell
/sig testing /sig scheduling
/assign


Question: "fmt" "math" "strings" <ins> "sync" </ins> "time"

What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this


Question: framework.ExpectNoError(err) for _, pod := range pods.Items { _, found := nodeToCapacityMap[pod.Spec.NodeName] <del> if found && pod.Status.Phase == api.PodRunning { </del> <ins> if found && pod.Status.Phase != api.PodSucceeded && pod.Status.Phase != api.PodFailed { </ins> framework.Logf("Pod %v requesting resource cpu=%vm on Node %v", pod.Name, getRequestedCPU(pod), pod.Spec.NodeName) nodeToCapacityMap[pod.Spec.NodeName] -= getRequestedCPU(pod) }

https://k8s- Failed: [] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance] {Kubernetes e2e suite}
- FYI
https://k8s- Failed: [] SchedulerPredicates [Serial] validates resource limits of pods that are allowed to run [Conformance] {Kubernetes e2e suite}
First failure is a dupe of . For second I need to wait for master logs.


Question: --storage-backend=${STORAGE_BACKEND} --service-cluster-ip-range="${SERVICE_CLUSTER_IP_RANGE}" EOF <ins> if [ -z "${CUSTOM_ADMISSION_PLUGINS:-}"]; then cat >> "${RESOURCE_DIRECTORY}/apiserver_flags" <<EOF --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota EOF else cat >> "${RESOURCE_DIRECTORY}/apiserver_flags" <<EOF --admission-control=${CUSTOM_ADMISSION_PLUGINS} EOF fi </ins> sed -i'' -e "s/"//g" "${RESOURCE_DIRECTORY}/apiserver_flags" cat > "${RESOURCE_DIRECTORY}/scheduler_flags" <<EOF

Until is fixed (which won't happen before code-freeze is finished), kubemark-scale will be failing/timing-out and this is "working-as-intended" until then. It should be possible to switch off secrets in kubemark to make other problems debuggable. For now we would like to: switch off secrets in kubemark-scale leave them in all other kubemark suites
can you add a release-blocking or non-blocking label to this issue?
Is kubemark scale useful/realistic if it disables things that are required in actual deployments?
We use it for development, and we need a way to switch off particular parts.
- it's just a short term fix that would allow to fix/debug different problems at the same time (we already know that secrets are causing problems and we will be fixing it).
To clarify - I'm not going to leave that forever - as soon as we will have some fix for secrets in, we will enable it back in 5000-node kubemark. Currently because of secrets those tests doesn't bring any value.
Also - this one is already done.


Question: configuration *atomic.Value lister        admissionregistrationlisters.ValidatingWebhookConfigurationLister hasSynced     func() bool <ins> // initialConfigurationSynced stores a boolean value, which tracks if // the existing webhook configs have been synced (honored) by the // manager at startup-- the informer has synced and either has no items // or has finished executing updateConfiguration() once. initialConfigurationSynced *atomic.Value </ins> } var _ generic.Source = &validatingWebhookConfigurationManager{}

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Testgrid link: Example pass: Example fail: Reason for failure: Unknown Anything else we need to know: Looks like it's an apiserver test, but I'm not too sure. links to appreciated: <!-- Please see the deflaking doc () for more guidance! --/kind flake
Looks like is flaky too. Example fail:
/sig api-machinery
Thanks for the report. Please report individual failing tests/packages so the responsible sigs can be assigned. The cm/devicemanager issue is tracked in
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I don't see individual issues yet, but I do see open PRs targeting this issue /remove-lifecycle rotten
closing in favor of specific issues: /close
Closing this issue. <detailsIn response to : closing in favor of specific issues: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "time" "github.com/onsi/ginkgo" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/intstr"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: // TODO consider printing more information about cluster } <del> func printService(out io.Writer, name, link string, warn bool) { </del> <ins> func printService(out io.Writer, name, link string) { </ins> ct.ChangeColor(ct.Green, false, ct.None, false) fmt.Fprint(out, name) ct.ResetColor()

Right now at the end of I get We should update the output to point to the appropriate v1beta3 URL endpoints instead.
/cc
Ref and . cc
And as a minor nit, not all of those URLs need a trailing slash hack e.g. I don't think it is required for Elasticsearch. I think most recently worked on this code?
Thanks for fixing this


Question: return } <ins> if f.preExecutePanic { panic("pre-exec-panic") } </ins> execFn() <ins> if f.postExecutePanic { panic("post-exec-panic") } </ins> f.lock.Lock() defer f.lock.Unlock()

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: } func TestNormalizeDesiredReplicasWithBehavior(t *testing.T) { <ins> now := time.Now() </ins> type TestCase struct { name                                string key                                 string

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: flags.StringVar(&TestContext.KubernetesAnywherePath, "kubernetes-anywhere-path", "/workspace/k8s.io/kubernetes-anywhere", "Which directory kubernetes-anywhere is installed to.") flags.BoolVar(&TestContext.ListImages, "list-images", false, "If true, will show list of images used for runnning tests.") <ins> flags.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") </ins> } // RegisterClusterFlags registers flags specific to the cluster e2e test suite.

Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes pod should support shared volumes between containers Since when has it been failing: Testgrid link: Reason for failure:
/sig testing
/assign
Thank for comment
Tests are now passing, /close
Closing this issue. <detailsIn response to : Tests are now passing, Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.ExpectEqual(*podStats.CPU.UsageCoreNanoSeconds > 0, true, "Pod stats should not report 0 cpu usage") framework.ExpectEqual(*podStats.Memory.WorkingSetBytes > 0, true, "Pod stats should not report 0 bytes for memory working set ") <ins> framework.ExpectEqual(podStats.Network != nil, true, "Pod stats should report network stats") framework.ExpectEqual(podStats.Network.Name != "", true, "Pod stats should report network name") framework.ExpectEqual(*podStats.Network.TxBytes > 0, true, "Pod stats should report network Tx stats") framework.ExpectEqual(len(podStats.Network.Interfaces) > 0, true, "Pod Stats should report individual interfaces stats") </ins> for _, containerStats := range podStats.Containers { framework.ExpectEqual(containerStats.Logs != nil, true, "Pod stats should have container log stats") framework.ExpectEqual(*containerStats.Logs.AvailableBytes > 0, true, "container log stats should not report 0 bytes for AvailableBytes")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When curling the endpoint for stats we don't see network stats. Network stats to be included when calling the summary. This works with dockershim for windows. This happens because Windows containerd uses hcsv2 to create containers and we query for the networks stats using hcsv1: This will be address eventually with move to CRI only metrics KEP: but is still several releases away. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: } } <del> func doTestPlugin(t *testing.T, spec *volume.Spec) { </del> <ins> func doTestPlugin(t *testing.T, spec *volume.Spec, expectedDevice string) { </ins> tmpDir, err := utiltesting.MkTmpdir("nfs_test") if err != nil { t.Fatalf("error creating temp dir: %v", err)

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: for feature, flag := range test.FeatureGates { defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } <del> perfScheduling(test, b) </del> <ins> dataItems.DataItems = append(dataItems.DataItems, perfScheduling(test, b)...) </ins> }) } <ins> if err := dataItems2JSONFile(dataItems, b.Name()); err != nil { klog.Fatalf("%v: unable to write measured data: %v", b.Name(), err) } </ins> } <del> func perfScheduling(test testCase, b *testing.B) { </del> <ins> func perfScheduling(test testCase, b *testing.B) []DataItem { </ins> var nodeStrategy testutils.PrepareNodeStrategy = &testutils.TrivialNodePrepareStrategy{} if test.Nodes.NodeAllocatableStrategy != nil { nodeStrategy = test.Nodes.NodeAllocatableStrategy

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, { <ins> desc: "valid input with ';' as separator for multiple reserved-memory arguments", argc: "blah --reserved-memory=0:memory=1Gi,hugepages-1Gi=1Gi;1:memory=1Gi", expectVal: []kubeletconfig.MemoryReservation{ { NumaNode: 0, Limits: v1.ResourceList{ v1.ResourceMemory:        memory1Gi, resourceNameHugepages1Gi: memory1Gi, }, }, { NumaNode: 1, Limits: v1.ResourceList{ v1.ResourceMemory: memory1Gi, }, }, }, }, { </ins> desc:      "invalid input", argc:      "blah --reserved-memory=bad-input", expectVal: nil,

I am trying out For that I enabled kubelet memory manager, cpu manager and topology manager. This is tried on a dual socket NUMA server KUBELETKUBEADMARGS="--container-runtime=remote --container-runtime- --cpu-manager-policy=static --feature-gates=RotateKubeletServerCertificate=true,AllAlpha=false,RemoveSelfLink=false,EphemeralContainers=true,BoundServiceAccountTokenVolume=false,KubeletPodResourcesGetAllocatable=true,TopologyManager=true,MemoryManager=true --kube-api-burst=40 --kube-api-qps=20 --kube-reserved=cpu=4,memory=4Gi --memory-manager-policy=static --reserved-cpus=0,1,2,3 --reserved-memory 0:memory=4Gi,1:memory=4196Mi --system-reserved=cpu=4,memory=4Gi --topology-manager-policy=single-numa-node" topologyManagerPolicy: single-numa-node is set on This throws an error 2021-10-20T11:28:49.+00:00 localhost kubelet[]: E1020 11:28:49. ] "Failed to parse kubelet flag" err="invalid argument "0:memory=4Gi,1:memory=4196Mi" for "--reserved-memory" flag: the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got 0:memory=4Gi,1:memory=4196Mi" This is on kubernetes 1.21.1 Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Any hints on specifying "reserved-memory" for multi-socket NUMA system ???? One way is specifying multiple "reserved-memory" like But problem with this is is map, so it will get over-written. we should be able to specify reserved memory in a multi socket system.

Enable above flags and run kubelet No response Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} none SLES 15 sp2 <details</detailscontainerd No response
/sig node
this is the right way to config
Problem with that is KubeletExtraArgs map[string]string is a map, so multiple "reserved-memory" will get over-written.
there is a tracking issue for kubeadm's api to support multiple arguments with the same name: but it is a backwards incompatible change. you should beusing the KubeletConfiguration for this instead:
/remove-kind support /kind bug /assign Looks like a bug in a parsing logic + better error would be beneficial
Is this a parsing bug?, if yes can you just point me where I need to fix this. I will try to fix this in the downstream release(our internal release).
Here the parsing logic - and here it is failing - Give me know if you need some additional help.
Are we planning to add support for multiple numa nodes in single "--reserved-memory" flag, or just displaying a better error msg?
We should support multiple NUMA nodes under the same --reserved-memory flag.
/triage accepted
Can you provide a sample configuration for 2 numa nodes? "." comma seperator is already there for single numa configuration.
Yes, it is the problem, initially we needed to split it by comma, but probably it will not work as expected, so maybe we can split it by and update documentation. Something like:
can you please review the PR and provide your comments


Question: - k8s.io/kubernetes/pkg/apis/core/helper - k8s.io/kubernetes/pkg/apis/core/install - k8s.io/kubernetes/pkg/apis/core/v1 <ins> - k8s.io/kubernetes/pkg/apis/discovery - k8s.io/kubernetes/pkg/apis/discovery/install - k8s.io/kubernetes/pkg/apis/discovery/v1alpha1 - k8s.io/kubernetes/pkg/apis/discovery/v1beta1 </ins> - k8s.io/kubernetes/pkg/apis/events - k8s.io/kubernetes/pkg/apis/events/install - k8s.io/kubernetes/pkg/apis/events/v1beta1 - k8s.io/kubernetes/pkg/apis/extensions - k8s.io/kubernetes/pkg/apis/extensions/install - k8s.io/kubernetes/pkg/apis/extensions/v1beta1 <ins> - k8s.io/kubernetes/pkg/apis/flowcontrol - k8s.io/kubernetes/pkg/apis/flowcontrol/install - k8s.io/kubernetes/pkg/apis/flowcontrol/v1alpha1 - k8s.io/kubernetes/pkg/apis/flowcontrol/v1beta1 - k8s.io/kubernetes/pkg/apis/imagepolicy - k8s.io/kubernetes/pkg/apis/imagepolicy/install - k8s.io/kubernetes/pkg/apis/imagepolicy/v1alpha1 </ins> - k8s.io/kubernetes/pkg/apis/networking <ins> - k8s.io/kubernetes/pkg/apis/node - k8s.io/kubernetes/pkg/apis/node/install - k8s.io/kubernetes/pkg/apis/node/v1 - k8s.io/kubernetes/pkg/apis/node/v1alpha1 - k8s.io/kubernetes/pkg/apis/node/v1beta1 </ins> - k8s.io/kubernetes/pkg/apis/policy - k8s.io/kubernetes/pkg/apis/policy/install - k8s.io/kubernetes/pkg/apis/policy/v1beta1

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: // - The second one runs grep 'hello world' on /mnt/test. // If both succeed, Kubernetes actually allocated something that is // persistent across pods. <del> By("checking the created volume is writable") runInPodWithVolume(client, claim.Namespace, claim.Name, "echo 'hello world' > /mnt/test/data") </del> <ins> By("checking the created volume is writable and has the PV's mount options") command := "echo 'hello world' > /mnt/test/data" // We give the first pod the secondary responsibility of checking the volume has // been mounted with the PV's mount options, if the PV was provisioned with any for _, option := range pv.Spec.MountOptions { // Get entry, get mount options at 6th word, replace brackets with commas command += fmt.Sprintf(" && ( mount | grep 'on /mnt/test' | awk '{print $6}' | sed 's/^(/,/; s/)$/,/' | grep -q ,%s, )", option) } runInPodWithVolume(client, claim.Namespace, claim.Name, command) </ins> By("checking the created volume is readable and retains data") runInPodWithVolume(client, claim.Namespace, claim.Name, "grep 'hello world' /mnt/test/data")

for 1.8. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
Looking at the PR description, it seems that adds tests for ReclaimPolicy only. What about MountOptions?
those are coming also, I'll re-rename issue/point to another one
Since the feature is in 1.8, we need tests in 1.8. Adding this to 1.8 milestone.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
This issue is a 1.8 blocker. The PRs for to fix it, and , are approved/LGTM'd and awaiting merge. This issue can be closed as soon as those merge.


Question: config.CurrentContext = o.contextName <del> if err := clientcmd.ModifyConfig(o.configAccess, *config, true); err != nil { return err } return nil </del> <ins> return clientcmd.ModifyConfig(o.configAccess, *config, true) </ins> } func (o *useContextOptions) complete(cmd *cobra.Command) error { endingArgs := cmd.Flags().Args() if len(endingArgs) != 1 { <del> cmd.Help() return fmt.Errorf("Unexpected args: %v", endingArgs) </del> <ins> return helpErrorf(cmd, "Unexpected args: %v", endingArgs) </ins> } o.contextName = endingArgs[0]

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: } if version == 0 { if !e.UpdateStrategy.AllowCreateOnUpdate() { <del> return nil, kubeerr.NewAlreadyExists(e.EndpointName, name) </del> <ins> return nil, kubeerr.NewNotFound(e.EndpointName, name) </ins> } creating = true if err := rest.BeforeCreate(e.CreateStrategy, ctx, obj); err != nil {

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: runTopologyManagerNodeAlignmentSuiteTests(f, sd, reservedSystemCPUs, policy, numaNodes, coreCount) } <del> // restore kubelet config setOldKubeletConfig(f, oldCfg) // Delete state file to allow repeated runs deleteStateFile() </del> }) ginkgo.It("run the Topology Manager pod scope alignment test suite", func() {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-node-kubelet-serial node-kubelet-serial job Which test(s) are failing: Hard to tell, but everything at one point or another. Since when has it been failing: A month or two, by recollection. would indicate several years... ci-kubernetes-node-kubelet-serial | failing_days | 992 Testgrid link: Reason for failure: Hard to tell, some is timeout, but I think timeouts due to test failures. Anything else we need to know: Nothing consistent about the failures that I can tell besides timeouts. This is the only place these serial tests are running.
/sig node
/cc
I will take a look at what happens with the serial lane.
So the kubelet failed to restart because of I think it can be related to the topology manager scope PR, I will dig into the code tomorrow. FYI
It actually looks like it was introduced by: With the same issue reported here: And Alexey has a fix for it here (I haven't reviewed it):
Thanks for the update.
Some of these failures are definitely due to , which is causing tests to fail since pods aren't terminating in time. See e.g.
/assign
Under the latest run, I saw that the kubelet failed to restart because of The problem that one of the topology manager tests failed in the middle and did not remove the state file, I can not guarantee that it a single problem, but let's start from it.


Question: } else { // We only share the refCounts for remote endpoints ep.refCount = proxier.endPointsRefCount.getRefCount(newHnsEndpoint.hnsID) <ins> *ep.refCount++ </ins> } ep.hnsID = newHnsEndpoint.hnsID <del> *ep.refCount++ </del> Log(ep, "Endpoint resource found", 3) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kube-proxy panics when a LoadBalancer service is created after a deployment it is associated with is created and the pods are fully running. kube-proxy does not panic. 1) Create 2 Windows nodes 2) Create a deployment, ensuring that the pods are scheduled on different nodes: 3) Wait for the pods to go into the state. kube-proxy will not crash if the pods are not fully deployed 4) Create a LoadBalancer service for the deployment 5) At this point kube-proxy should have crashed. If not, delete and recreate the LoadBalancer, this should trigger the panic. The kube-proxy exe was run as: Stdout/err shows: Full kube-proxy logs: Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: AWS OS (e.g: ): Windows server 2019 Kernel (e.g. ): 10.0..1817 Install tools: OpenShift 4.7.3 Network plugin and version (if this is a network-related bug): Hybrid OVN Others:
/sig windows
are you aware of anything happening here?
I am taking a look at this.
/triage accepted


Question: name: coredns namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: <del> replicas: 2 </del> <ins> # replicas: not specified here: # 1. In order to make Addon Manager do not reconcile this replicas parameter. # 2. Default is 1. # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on. </ins> strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> template: metadata: labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> spec: serviceAccountName: coredns tolerations:

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: panic(err) } } <ins> func convert_v1_StatusDetails_To_api_StatusDetails(in *StatusDetails, out *api.StatusDetails, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*StatusDetails))(in) } out.ID = in.Name out.Kind = in.Kind if in.Causes != nil { out.Causes = make([]api.StatusCause, len(in.Causes)) for i := range in.Causes { if err := convert_v1_StatusCause_To_api_StatusCause(&in.Causes[i], &out.Causes[i], s); err != nil { return err } } } else { out.Causes = nil } out.RetryAfterSeconds = in.RetryAfterSeconds return nil } func convert_api_StatusDetails_To_v1_StatusDetails(in *api.StatusDetails, out *StatusDetails, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*api.StatusDetails))(in) } out.Name = in.ID out.Kind = in.Kind if in.Causes != nil { out.Causes = make([]StatusCause, len(in.Causes)) for i := range in.Causes { if err := convert_api_StatusCause_To_v1_StatusCause(&in.Causes[i], &out.Causes[i], s); err != nil { return err } } } else { out.Causes = nil } out.RetryAfterSeconds = in.RetryAfterSeconds return nil } func convert_v1_StatusCause_To_api_StatusCause(in *StatusCause, out *api.StatusCause, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*StatusCause))(in) } out.Type = api.CauseType(in.Type) out.Message = in.Message out.Field = in.Field return nil } func convert_api_StatusCause_To_v1_StatusCause(in *api.StatusCause, out *StatusCause, s conversion.Scope) error { if defaulting, found := s.DefaultingInterface(reflect.TypeOf(*in)); found { defaulting.(func(*api.StatusCause))(in) } out.Type = CauseType(in.Type) out.Message = in.Message out.Field = in.Field return nil } </ins>

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: "Rev": "6633656539c1639d9d78127b7d47c622b5d7b6dc" }, { <ins> "ImportPath": "github.com/influxdb/influxdb/client", "Comment": "v0.8.8", "Rev": "afde71eb1740fd763ab9450e1f700ba0e53c36d0" }, { </ins> "ImportPath": "github.com/kr/pty", "Comment": "release.r56-25-g05017fc", "Rev": "05017fcccf23c823bfdea560dcc958a136e54fb7"

Due to changes in field selectors (see ) heapster was almost certainly busted, but continues to pass. This seems incorrect.