Question: } return true } <ins> // GetAllLocalAddressesExcept return all local addresses on the node, // except from the passed dev.  This is not the same as to take the // diff between GetAllLocalAddresses and GetLocalAddresses since an // address can be assigned to many interfaces. This problem raised // https://github.com/kubernetes/kubernetes/issues/114815 func (h *netlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { ifaces, err := net.Interfaces() if err != nil { return nil, err } var addr []net.Addr for _, iface := range ifaces { if iface.Name == dev { continue } ifadr, err := iface.Addrs() if err != nil { // This may happen if the interface was deleted. Ignore // but log the error. klog.ErrorS(err, "Reading addresses", "interface", iface.Name) continue } addr = append(addr, ifadr...) } return utilproxy.AddressSet(h.isValidForSet, addr), nil } </ins>

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: if err != nil { return err } <del> unsupportedCoreDNS, err := migration.Unsupported(currentInstalledCoreDNSversion, kubeadmconstants.CoreDNSVersion, corefile) </del> <ins> unsupportedCoreDNS, err := migration.Unsupported(currentInstalledCoreDNSversion, currentInstalledCoreDNSversion, corefile) </ins> if err != nil { return err } if unsupportedCoreDNS != nil { <del> var UnsupportedPlugins, UnsupportedVersion string </del> <ins> var UnsupportedPlugins []string </ins> for _, unsup := range unsupportedCoreDNS { <del> UnsupportedPlugins = unsup.Plugin UnsupportedVersion = unsup.Version } if UnsupportedPlugins != "" || UnsupportedVersion != "" { return errors.New("there are unsupported plugins in the CoreDNS Corefile") </del> <ins> UnsupportedPlugins = append(UnsupportedPlugins, unsup.ToString()) </ins> } <ins> fmt.Println("[preflight] The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. " + "Each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. " + "Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, " + "kubeadm will migrate the configuration while leaving the listed plugin configs untouched, " + "but cannot guarantee that they will work with the newer version of CoreDNS.") return errors.Errorf("CoreDNS cannot migrate the following plugins:n%s", UnsupportedPlugins) </ins> } return nil }

Any ideas how to solve this?
What does your look like?
/assign
I see that the plugin is present in the config. From what version of Kubernetes are you upgrading from?
I was on 1.15.3 and now trying upgrading to 1.15.4 first. Edit: I'm now on 1.15.4 but upgrade doesn't work to 1.16.0. I don't know what proxy plugin is... must be something that is comming automatic.
The preflight check is showing due to the presence of the plugin, as it has been deprecated. The plugin is to be used in its place. If the is the same as you've mentioned , you can use the while upgrading. The plugin will be replaced to use automatically.
Worked for me. Looks like all services are back up :-) Thanks!
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Having the same issue here. I installed clean k8s 1.13.4 , didn't install or anything. Just upgraded to 1.14.7, then to 1.15.4 and now see the same issue trying to upgrade to 1.16.0
Found and follow ups. removed from configMap kube-system/coredns no more complaints
Seems, I was a little bit to fast. After upgrading I saw a lot of errors in the coredns POD logs and (external) DNS resolution didn't work at all: Then I : Forwarding all requests to the nameservers in the nodes So the solution is replace in : with: see also:
I am having a similar issue, but this is related to rewrite plugin, which I believe is supported in the newer coredns version as well. I can workaround by skipping (or removing rewrite rule and adding it back again). Let me know if you want me to create a new issue to track this, or if this is already tracked. Kubernetes upgrade from 1.15.3 to 1.16.2 kubeadm upgrade plan coreFile
Yes, the upgrade should happen smoothly by just skipping the preflight check.

I’ll improve the functionality of the preflight check since more than a few users are having trouble with this. /reopen
Reopened this issue. <detailsIn response to : Yes, the upgrade should happen smoothly by just skipping the preflight check. I’ll improve the functionality of the preflight check since more than a few users are having trouble with this. /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area kubeadm /sig cluster-lifecycle
Hi , I have the same issue here due to the plugin of coredns that is reported as unsupported. My coredns config: Result of Kubadm version: If I change my coredns configuration to remove usage of plugin, the upgrade plan works: I don't think plugin is unsupported by Coredns, unless I'm missing something.
Hosts is not currently supported by the . However, the presence of a plugin that is not supported by migration library should not be preventing the kubeadm upgrade from proceeding if you use the flag, per .
+1 Lets treat ignoring the error as the workaround users should do, for this unsupported scenario. /close
Closing this issue. <detailsIn response to : +1 Lets treat ignoring the error as the workaround users should do, for this unsupported scenario. </details
Sounds good. FWIW, The original reasoning for requiring the explicit ignoring of the pre-flight check was to make the user aware of the risk that proceeding with the upgrade could result in an invalid Corefile. So, in that sense, its working as intended, but the error message is perhaps too cryptic. Currently: "There are unsupported plugins in the CoreDNS Corefile" What was intended to be conveyed: "There are plugin configurations in the CoreDNS Corefile that cannot be migrated. Continuing with the upgrade without migrating those plugin configurations could result in an invalid Corefile and therefore DNS interruption."
I think improving the message is fine or even enumerating which plugins.
Why was this closed? As I understand the issue, kubeadm fails to upgrade without additional options because of a configuration that was created by a previous kubeadm?
i don't believe this part is true.

the "hosts" plugin is not part of the kubeadm 1.16 manifests, so it's user :
hosts is, but proxy was in the default:
the proxy plugin is mentioned as deprecated:
I know, but ignoring the preflight warning makes the upgrade go through just fine, including migration the ConfigMap. So what is the point of the warning?
I think the migration logic in kubeadm checks the supportability of the current config in all versions from the starting point to the destination - hence kubeadm was alerting incorrectly about proxy, which would have been valid in the current version, but then deprecated/removed by the destination version. It should instead check only the starting point - ensuring that there are no unsupported plugins present in the start version.
please reopen this issue if you think there are PRs that we should sent.
Yes after looking at the related code, it looks like kubeadm uses a "dry run" call to in the preflight check to see if there are any unsupported plugins. It should probably instead call , for the starting version only. /reopen
Reopened this issue. <detailsIn response to : Yes after looking at the related code, it looks like kubeadm uses a "dry run" call to in the preflight check to see if there are any unsupported plugins. It should probably instead call , for the starting version only. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Something didn't seem right about my explanation above so I took a second look - looks like there are two preflight checks. One dry run call to , which is OK as is. There is another preflight check, that calls . It is called with both the starting version and destination version, which checks if the existing config contains any plugins that are unsupported in any version from start to destination. Instead it should be called with only the starting version.
Totally agree with that. When I got the error, I did not know what to do. I had to go on google and hopefully found this post. I removed "proxy" module myself in configuration file, but the issue persisted. I had to look at the code to understand why I got this error. This is not something I should need to do.

Providing the list of unsuported modules would be a good starting point, because we would know why the preflight check is failing. Also, a more explicit error message would be great as well. If I understand well, the message could be something like this: "There are plugin in the CoreDNS Corefile that are not supported by Kubeadm or are deprecated in the next release. Please, review your CoreDNS configuration against Changelog to ensure all your plugins are supported by the new version of CoreDNS" (English is not my native language, so you may found better formulation).
Thanks, Just to summarize, there are two issues raised here: is a in kubeadm's CoreDNS configuration migration preflight check. When the proxy plugin is in use, the preflight check falsely alerts that the proxy plugin is not supported by the configuration migration when in fact it is - the migration will automatically replace proxy with the forward plugin. The workaround for this issue now is to ignore the false warning - i.e. skip the preflight check. plugins that are actually not supported by the configuration migrator (e.g. hosts for now), the warning is cryptic and should be expanded/rephrased to more clearly convey the following: The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. A list of those plugins should be provided in the preflight check message. To be safe, each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, kubeadm will migrate the configuration while leaving the listed plugin configs untouched, but cannot guarantee that they will work with the newer version of CoreDNS. Perhaps a link to more detailed documentation if it gets too hard to succinctly explain this (could go in coredns/corefile-migration).


Question: if es.cgroupsPerQOS { cmdArgs = append(cmdArgs, "--cgroups-per-qos", "true", <del> "--cgroup-root", "/", </del> ) } if !*disableKubenet {

In node-e2e-test on Containervm image, the containers of the pods created by Kubernetes are subcontainers of docker-daemon/docker/. It means that the runtime resource usage (docker-daemon) is not correct, because all pods are included. This problem maybe due to container misconfiguration in node-e2e-test.
The kubelet on debian-based containervm image is misconfigured. Will fix it shortly.


Question: message Mount { // Path of the mount within the container. string container_path = 1; <del> // Path of the mount on the host. </del> <ins> // Path of the mount on the host. If the hostPath doesn't exist, then runtimes // should report error. If the hostpath is a symbolic link, runtimes should // follow the symlink and mount the real destination to container. </ins> string host_path = 2; // If set, the mount is read-only. bool readonly = 3;

Currently, Docker create the host directory if the host path does not exist. For host path, Kubelet doesn't check the host directory existence, and pass the mount to docker directly. Given so, we should: 1) Either clearly define the expectation in CRI; 2) Or create the host directory ourselves when it doesn't exist. /cc
I think hostPath is expected to be already exist and kubelet should create the host directory if it is not exist.
Hmm, another problem is how to known whether hostPath is (expected) file or directory? Since it doesn't exist, we can only assume it is a directory?
There is a new field in the hostpath spec, that may help:
Another thing we may want to clearly define is: If when the host path is a symlink, what is the expected behavior. I think we assume container runtime should follow the symlink and mount the real content, but it's not clearly defined anywhere.
can you clarify, when you say "host path", are you referring specifically to hostPath volumes, or to any volume plugin's host-side path that gets passed into CRI?
I believe it's .
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "strconv" "strings" "testing" <ins> "time" </ins> "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-06-01/network" "github.com/Azure/go-autorest/autorest/to"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The issue has been observed on Azure but should also apply to other providers as well. On Azure we observed that there are orphaned Public IPs that don't belong to any Kubernetes LoadBalancer service but from looking at the tags and the name it is clear that Kubernetes created them. In the Azure Activity Log, you can see that the Public IP was deleted when the K8s service was deleted but recreated a few minutes later. That re-creation is also visible in the K8s controller manager logs as a reconcile of the service. In our case, we add additional worker nodes and delete services at the same time. I think this leads to the re-creation here Because the service list is retrieved at the beginning of the reconcile loop, services that are deleted while the loop is reconciling a recreated by that loop and afterwards orphaned. What you expected to happen: No orphaned public IPs How to reproduce it (as minimally and precisely as possible): Create a few Kubernetes Load Balancer services (we have around 20). Then scale your worker nodes and delete one service at the same time. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: Azure OS (e.g: ): Core OS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
Thanks for reporting the issue. I think this should be fixed during v1.16 release cycle. /milestone v1.16
In the nodeSyncLoop -updateLoadBalancerHosts -lockedUpdateLoadBalancerHosts call chain, I don't see where the mutex lock is obtained.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 4 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged.

Is this issue still planned to be fixed for the 1.16 release?
The issue would probably not able to be fixed in v1.16. AFAIK, no one is working on it now.
Thank you for the update! I'll go ahead and remove the issue from the milestone. If anything changes in the meanwhile, please let us know. /milestone clear
/assign
/assign
GitHub didn't allow me to assign the following users: chewong. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't LB Finalizer support fix this? We enabled this by default starting v1.16
Initial PR: Bump to beta in v1.16:


Question: ginkgo.BeforeEach(func() { c = f.ClientSet ns = f.Namespace.Name <ins> dc = f.DynamicClient </ins> }) ginkgo.It("deployment reaping should cascade to its replica sets and pods", func() {

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: t.Errorf("expected enabled for %v, from %v", g2v1rEnabled, config) } <del> // Enable all enables specific resources </del> <ins> // DisableAll() only disables to the group/version level for compatibility // corresponds to --runtime-config=api/all=false config.DisableAll() if config.ResourceEnabled(g1v1rEnabled) { t.Errorf("expected disabled for %v, from %v", g1v1rEnabled, config) } if config.ResourceEnabled(g1v2rEnabled) { t.Errorf("expected disabled for %v, from %v", g1v2rEnabled, config) } if config.ResourceEnabled(g2v1rEnabled) { t.Errorf("expected disabled for %v, from %v", g2v1rEnabled, config) } // DisableAll() only disables to the group/version level for compatibility // corresponds to --runtime-config=api/all=false,g1/v1=true config.DisableAll() config.EnableVersions(g1v1) if !config.ResourceEnabled(g1v1rEnabled) { t.Errorf("expected enabled for %v, from %v", g1v1rEnabled, config) } // EnableAll() only enables to the group/version level for compatibility </ins> config.EnableAll() <del> // all resources under g1v1 are now enabled </del> <ins> // all unspecified or enabled resources under all groups now enabled </ins> if !config.ResourceEnabled(g1v1rUnspecified) { t.Errorf("expected enabled for %v, from %v", g1v1rUnspecified, config) } if !config.ResourceEnabled(g1v1rEnabled) { t.Errorf("expected enabled for %v, from %v", g1v1rEnabled, config) } <del> if !config.ResourceEnabled(g1v1rDisabled) { t.Errorf("expected enabled for %v, from %v", g1v1rDisabled, config) </del> <ins> if !config.ResourceEnabled(g1v2rUnspecified) { t.Errorf("expected enabled for %v, from %v", g1v2rUnspecified, config) } if !config.ResourceEnabled(g1v2rEnabled) { t.Errorf("expected enabled for %v, from %v", g1v2rEnabled, config) } if !config.ResourceEnabled(g2v1rUnspecified) { t.Errorf("expected enabled for %v, from %v", g2v1rUnspecified, config) } if !config.ResourceEnabled(g2v1rEnabled) { t.Errorf("expected enabled for %v, from %v", g2v1rEnabled, config) </ins> } <del> // previously disabled resources are now enabled if !config.ResourceEnabled(g1v2rDisabled) { t.Errorf("expected enabled for %v, from %v", g1v2rDisabled, config) </del> <ins> // previously disabled resources are still disabled if config.ResourceEnabled(g1v1rDisabled) { t.Errorf("expected disabled for %v, from %v", g1v1rDisabled, config) </ins> } <del> if !config.ResourceEnabled(g2v1rDisabled) { t.Errorf("expected enabled for %v, from %v", g2v1rDisabled, config) </del> <ins> if config.ResourceEnabled(g1v2rDisabled) { t.Errorf("expected disabled for %v, from %v", g1v2rDisabled, config) } if config.ResourceEnabled(g2v1rDisabled) { t.Errorf("expected disabled for %v, from %v", g2v1rDisabled, config) </ins> } }

https://k8s- sample failure: (also affecting other CI jobs) cc /sig gcp
/assign We're interested in disabling the job as blocking entirely. We're of the opinion this shouldn't have bee allowed to block the community in the first place. I swear there's a historical issue around tightening the regex somewhere.
... wrong button sorry about that
possibly caused by erroring on invalid values that were previously silently skipped opened to downgrade that error to a warning for a release
Well, I mentioned the idea of making the job optional in
PR out to drop the runifchanged regex from pull-kubernetes-e2e-gke
Confirmed turns the test green
Can call this closed when merges
/pony hooray
Couldn't find a pony matching that query. <detailsIn response to : /pony hooray Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return status } <del> // Make the latest container status comes first. sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) </del> <ins> // Fetch old containers statuses from old pod status. oldStatuses := make(map[string]api.ContainerStatus, len(pod.Spec.Containers)) for _, status := range pod.Status.ContainerStatuses { oldStatuses[status.Name] = status } // Set all container statuses to default waiting state </ins> statuses := make(map[string]*api.ContainerStatus, len(pod.Spec.Containers)) <del> // Create a map of expected containers based on the pod spec. expectedContainers := make(map[string]api.Container) </del> <ins> defaultWaitingState := api.ContainerState{Waiting: &api.ContainerStateWaiting{Reason: "ContainerCreating"}} </ins> for _, container := range pod.Spec.Containers { <del> expectedContainers[container.Name] = container </del> <ins> status := &api.ContainerStatus{ Name:  container.Name, Image: container.Image, State: defaultWaitingState, } // Apply some values from the old statuses as the default values. if oldStatus, found := oldStatuses[container.Name]; found { status.RestartCount = oldStatus.RestartCount status.LastTerminationState = oldStatus.LastTerminationState } statuses[container.Name] = status </ins> } <del> containerDone := sets.NewString() apiPodStatus.PodIP = podStatus.IP for _, containerStatus := range podStatus.ContainerStatuses { cName := containerStatus.Name if _, ok := expectedContainers[cName]; !ok { </del> <ins> // Make the latest container status comes first. sort.Sort(sort.Reverse(kubecontainer.SortContainerStatusesByCreationTime(podStatus.ContainerStatuses))) // Set container statuses according to the statuses seen in pod status containerSeen := map[string]int{} for _, cStatus := range podStatus.ContainerStatuses { cName := cStatus.Name if _, ok := statuses[cName]; !ok { </ins> // This would also ignore the infra container. continue } <del> if containerDone.Has(cName) { </del> <ins> if containerSeen[cName] >= 2 { </ins> continue } <del> status := convertContainerStatus(containerStatus) if existing, found := statuses[cName]; found { existing.LastTerminationState = status.State containerDone.Insert(cName) } else { </del> <ins> status := convertContainerStatus(cStatus) if containerSeen[cName] == 0 { </ins> statuses[cName] = status <ins> } else { statuses[cName].LastTerminationState = status.State </ins> } <ins> containerSeen[cName] = containerSeen[cName] + 1 </ins> } <del> // Handle the containers for which we cannot find any associated active or dead containers or are in restart backoff // Fetch old containers statuses from old pod status. // TODO(random-liu) Maybe it's better to get status from status manager, because it takes the newest status and there is not // status in api.Pod of static pod. oldStatuses := make(map[string]api.ContainerStatus, len(pod.Spec.Containers)) for _, status := range pod.Status.ContainerStatuses { oldStatuses[status.Name] = status } </del> <ins> // Handle the containers failed to be started, which should be in Waiting state. </ins> for _, container := range pod.Spec.Containers { <del> // TODO(random-liu): We should define "Waiting" state better. And cleanup the following code. if containerStatus, found := statuses[container.Name]; found { reason, message, ok := kl.reasonCache.Get(uid, container.Name) if ok && reason == kubecontainer.ErrCrashLoopBackOff { containerStatus.LastTerminationState = containerStatus.State containerStatus.State = api.ContainerState{ Waiting: &api.ContainerStateWaiting{ Reason:  reason.Error(), Message: message, }, } } </del> <ins> // If a container should be restarted in next syncpod, it is *Waiting*. if !kubecontainer.ShouldContainerBeRestarted(&container, pod, podStatus) { </ins> continue } <del> var containerStatus api.ContainerStatus containerStatus.Name = container.Name containerStatus.Image = container.Image if oldStatus, found := oldStatuses[container.Name]; found { // Some states may be lost due to GC; apply the last observed // values if possible. containerStatus.RestartCount = oldStatus.RestartCount containerStatus.LastTerminationState = oldStatus.LastTerminationState } reason, _, ok := kl.reasonCache.Get(uid, container.Name) </del> <ins> status := statuses[container.Name] reason, message, ok := kl.reasonCache.Get(uid, container.Name) </ins> if !ok { <del> // default position for a container // At this point there are no active or dead containers, the reasonCache is empty (no entry or the entry has expired) // its reasonable to say the container is being created till a more accurate reason is logged containerStatus.State = api.ContainerState{ Waiting: &api.ContainerStateWaiting{ Reason:  fmt.Sprintf("ContainerCreating"), Message: fmt.Sprintf("Image: %s is ready, container is creating", container.Image), }, } } else if reason == kubecontainer.ErrImagePullBackOff || reason == kubecontainer.ErrImageInspect || reason == kubecontainer.ErrImagePull || reason == kubecontainer.ErrImageNeverPull || reason == kubecontainer.RegistryUnavailable { // mark it as waiting, reason will be filled bellow containerStatus.State = api.ContainerState{Waiting: &api.ContainerStateWaiting{}} } else if reason == kubecontainer.ErrRunContainer { // mark it as waiting, reason will be filled bellow containerStatus.State = api.ContainerState{Waiting: &api.ContainerStateWaiting{}} </del> <ins> // In fact, we could also apply Waiting state here, but it is less informative, // and the container will be restarted soon, so we prefer the original state here. // Note that with the current implementation of ShouldContainerBeRestarted the original state here // could be: //   * Waiting: There is no associated historical container and start failure reason record. //   * Terminated: The container is terminated. continue } if status.State.Terminated != nil { status.LastTerminationState = status.State </ins> } <del> statuses[container.Name] = &containerStatus </del> <ins> status.State = api.ContainerState{ Waiting: &api.ContainerStateWaiting{ Reason:  reason.Error(), Message: message, }, } statuses[container.Name] = status </ins> } apiPodStatus.ContainerStatuses = make([]api.ContainerStatus, 0) <del> for containerName, status := range statuses { if status.State.Waiting != nil { status.State.Running = nil // For containers in the waiting state, fill in a specific reason if it is recorded. if reason, message, ok := kl.reasonCache.Get(uid, containerName); ok { status.State.Waiting.Reason = reason.Error() status.State.Waiting.Message = message } } </del> <ins> for _, status := range statuses { </ins> apiPodStatus.ContainerStatuses = append(apiPodStatus.ContainerStatuses, *status) }

We moved reason cache and status convert function from to by , , and . After this, we should also migrate related old unit test to kubelet level. For we should add test for: [x] Different image pull policy. [x] Pod creation failure. [x] Pod image failure. For we should add test for: [x] ContainerStatuses order. [x] Last termination state. [x] How reason cache will affect the PodPhase: [x] Pull image failure reason [x] Pod creation failure reason
Time to close this issue? :)
Not yet, The last 3 tests are in . :)


Question: file, err := os.Open(dataFileName) if err != nil { <del> return nil, errors.New(log("failed to open volume data file [%s]: %v", dataFileName, err)) </del> <ins> return nil, fmt.Errorf("%s: %w", log("failed to open volume data file [%s]", dataFileName), err) </ins> } defer file.Close() data := map[string]string{}

Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted.

So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation.

This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node .

when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ]

"Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version


Question: "strings" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" api "k8s.io/kubernetes/pkg/apis/core" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In k8s 1.19.2 If the seccomp annotations of psp doesn't contain , pod with seccomp annotations would not start with error: psp with works psp without doesn't work Is this due to ? What you expected to happen: In k8s 1.18, psp annotations without works. I think in k8s 1.19, it should also work.
/sig auth
/assign I think this was a regression with the seccomp graduation to GA. Since we auto-convert 'docker/default' to the runtime default in fields, but PSPs that only allow 'docker/default' aren't accepting 'runtime/default'. I think the easiest fix is to just add both and to the allowed profiles map whenever either is set (here: ) This will need to be cherry-picked /priority important-soon
I'm happy to work on this one tomorrow if is not tackling the issue earlier :upsidedownface:
Fix is incoming in
Thanks for the quick fix ! Can you open a cherry-pick for v1.19 too? (see )
Forgot to link it, but this is already done: :)
Is there an easy way to check in which version of k8s this fix is applied? Label, milestone, etc? Another question, what might be a workaround without updating cluster?
Find the fix commit and run And there's also a cherry-pick commit: The workaround could be add annotations to your PSPs, take a look at this PR:


Question: } } <del> s.Broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: s.EventClient.Events("")}) </del> <ins> if s.Broadcaster != nil && s.EventClient != nil { s.Broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: s.EventClient.Events("")}) } </ins> // Start up a healthz server if requested if s.HealthzServer != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug What happened: Got panic after running: Panic: What you expected to happen: No panic and exit successfully. How to reproduce it (as minimally and precisely as possible): Run the following command on both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Happened with the following versions: v1.7.0-alpha.3 v1.7.0-alpha.4 v1.7.0-beta.0 v1.7.0-beta.1 v1.7.0-beta.2 v1.7.0-rc.1 v1.8.0-alpha.0 v1.8.0-alpha.1 Anything else we need to know?: I guess it caused by . Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Kernel (e.g. ): CoreOS v1298.7.0: Linux node- 4.9.16-coreos-r1 SMP Tue Apr 4 19:09:38 UTC 2017 x8664 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz GenuineIntel GNU/Linux Ubuntu 16.04.2 LTS: Linux ubuntu- 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network /area kube-proxy
/cc
cc/ Looks like there is a new regression in v1.7. Please triage this to decide if it is a release blocker. If not, please provide the workaround for the users. Thanks!

cc/
Digging into the code, will send a PR.
PR sent. Folks review it please.
Added the release-blocker label. We got a PR for it very fast so it won't delay the release anyway, thanks


Question: // NodeTree is a fake method for testing. func (c *Cache) NodeTree() *internalcache.NodeTree { return nil } <ins> // GetNodeInfo is a fake method for testing. func (c *Cache) GetNodeInfo(nodeName string) (*v1.Node, error) { return nil, nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution.

After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.


Question: testCases := map[string]struct { clientConfig *testClientConfig icc          *testICC <ins> overrides    *ConfigOverrides </ins> checkedICC bool result     string <del> ok         bool </del> <ins> overridden bool </ins> err        error }{ "in-cluster checked on empty error": {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In-cluster namespace isn't overridden by kubeconfig if the context is specified via kubectl flag. When using , the override works as expected. What you expected to happen: If I specify for a context in kubeconfig, then I expect to use as the default namespace, regardless of whatever's in . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: supposedly fixed this issue, but not for the case where the context is specified via a command line flag (). This is because this line checks for . Environment: Kubernetes version (use ): v1.10.5 (details above) Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): - Kernel (e.g. ): - Install tools: - Others: - <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in


Question: # A set of helpers for starting/running etcd for tests <del> ETCD_VERSION=${ETCD_VERSION:-3.5.3} </del> <ins> ETCD_VERSION=${ETCD_VERSION:-3.5.4} </ins> ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-2379} # This is intentionally not called ETCD_LOG_LEVEL:

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: func (e *exists) Admit(a admission.Attributes) (err error) { defaultVersion, kind, err := api.RESTMapper.VersionAndKindForResource(a.GetResource()) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } mapping, err := api.RESTMapper.RESTMapping(kind, defaultVersion) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } if mapping.Scope.Name() != meta.RESTScopeNameNamespace { return nil

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: configFile = "config/performance-config.yaml" ) <ins> var ( defaultMetrics = []string{ "scheduler_scheduling_algorithm_predicate_evaluation_seconds", "scheduler_scheduling_algorithm_priority_evaluation_seconds", "scheduler_binding_duration_seconds", "scheduler_e2e_scheduling_duration_seconds", } ) </ins> // testCase configures a test case to run the scheduler performance test. Users should be able to // provide this via a YAML file. //

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.Failf("%v", err) } }) <ins> // Marked as flaky until https://github.com/kubernetes/kubernetes/issues/65517 is solved. ginkgo.It("[Flaky] kubectl explain works for CR with the same resource name as built-in object.", func() { customServiceShortName := fmt.Sprintf("ksvc-%d", time.Now().Unix()) // make short name unique opt := func(crd *apiextensionsv1.CustomResourceDefinition) { crd.ObjectMeta = metav1.ObjectMeta{Name: "services." + crd.Spec.Group} crd.Spec.Names = apiextensionsv1.CustomResourceDefinitionNames{ Plural:     "services", Singular:   "service", ListKind:   "ServiceList", Kind:       "Service", ShortNames: []string{customServiceShortName}, } } crdSvc, err := setupCRDAndVerifySchemaWithOptions(f, schemaCustomService, schemaCustomService, "service", []string{"v1"}, opt) if err != nil { framework.Failf("%v", err) } if err := verifyKubectlExplain(f.Namespace.Name, customServiceShortName+".spec", `(?s)DESCRIPTION:.*Specification of CustomService.*FIELDS:.*dummy.*<string>.*Dummy property`); err != nil { _ = cleanupCRD(f, crdSvc) // need to remove the crd since its name is unchanged framework.Failf("%v", err) } if err := cleanupCRD(f, crdSvc); err != nil { framework.Failf("%v", err) } }) </ins> }) func setupCRD(f *framework.Framework, schema []byte, groupSuffix string, versions ...string) (*crd.TestCrd, error) {

Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] Since when has it been failing: 04-26 05:02 PDT Testgrid link: Reason for failure: Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: stderr: the server doesn't have a resource type "ksvc" error: exit status 1 Full Stack Trace () +0x1545 (0xc002ca6100) +0x324 (0xc002ca6100) +0x2b testing.tRunner(0xc002ca6100, 0x4b8d060) +0xc9 created by testing.(T).Run +0x350 [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. Anything else we need to know: /cc /cc /milestone v1.19 /priority critical-urgent
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

<detailsIn response to : Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] 04-26 05:02 PDT Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: the server doesn't have a resource type "ksvc" exit status 1 () (0xc002ca6100) (0xc002ca6100) testing.tRunner(0xc002ca6100, 0x4b8d060) created by testing.(T).Run [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. /cc /milestone v1.19 </details
/milestone v1.19 Thanks for opening this! I'll get you on milestone maintainers soon :) Also, was taking a look at this as well and noticed this test was recently :
As noted in , this modified an existing conformance test and made it flaky. Please move the new test out of the conformance test and mark it as flaky until this issue has been resolved.

Cc for flake introduced in
It is a bit weird that I cannot reproduce this problem on my local kind cluster by running even after 20 times I suspect this is a cache issue since I once created a crd without short name and forgot to delete it in the test.
/retitle [Flaky Test]: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) /kind flake /remove-kind failing-test
/assign
I can reproduce this problem by the following steps now. It looks like that this problem has something to do with discovery cache. <pre$ cat <summary<detailsapiVersion: kind: CustomResourceDefinition metadata: name: spec: conversion: strategy: None group: crd-publish-openapi-test- names: kind: Service listKind: ServiceList plural: services singular: service shortNames: ["ksvc"] scope: Namespaced versions: name: v1alpha1 schema: openAPIV3Schema: properties: spec: description: Specification of CustomService properties: dummy: description: Dummy property. type: string type: object type: object served: true storage: true </details</summary$ kubectl get crd No resources found in default namespace. $ rm -r $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl create -f created $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl explain <summary<detailsKIND: Service VERSION: crd-publish-openapi-test- RESOURCE: spec <ObjectDESCRIPTION: Specification of CustomService FIELDS: dummy <stringDummy property. </details</summary</pre
looks like
/assign


Question: // can't delete a target pool that's currently in use by a forwarding rule. // Thus, we have to tear down the forwarding rule if either it or the target // pool needs to be updated. <del> if fwdRuleExists && (fwdRuleNeedsUpdate || tpNeedsUpdate) { </del> <ins> if fwdRuleExists && (fwdRuleNeedsUpdate || tpNeedsRecreation) { </ins> // Begin critical section. If we have to delete the forwarding rule, // and something should fail before we recreate it, don't release the // IP.  That way we can come back to it later.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: t.Error(err) } } <ins> // Expecting Exists func to throw error if VM deleted _, err = vm.Exists(ctx) if err == nil { t.Error("expected error") } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have k8s cluster with version . And I noticed that if I deleted worker node VM from IAAS, then node stuck in state, I created statefulset before and the pod scheduled on that node was stuck in state, and no new pod was created to replace that. Node should be deleted if no instance is found Related log When I check the log of I could see following log keeps popping up. Looks like node_controller noticed that the VM is gone but it doesn't delete the node. Related code here will return if no instance is found. But it should check if it's VM is deleted not shutdown, then return And it should cherry pick this fix avoiding deleting shutdown node What you expected to happen: After VM is deleted, the node should be deleted, pods assigned to the node should be rescheduled to other available nodes How to reproduce it (as minimally and precisely as possible): k8s cluster. In our case we use PKS. workloads on node worker node VM from IAAS Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig cloud-provider /sig vmware
Same issue as this one The issue and is cluster API repo. We should fix it inside node_controller in k/k
Hi this issue was found when testing windows cluster deployed using PKS. We do have a workaround by manually deleting node using kubectl. Still we wish the fix could go in ASAP. Could you ask someone from upstream to to review the PR that fix this issue? Thank you
, we will try to push this through for v1.20. 1.19 is approaching code thaw, so we can't push this through
Thanks Michael. I'll let my team know.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: ) var ( <del> encryptionConfigAutomaticReloadFailureTotal = metrics.NewCounterVec( </del> <ins> encryptionConfigAutomaticReloadsTotal = metrics.NewCounterVec( </ins> &metrics.CounterOpts{ Namespace:      namespace, Subsystem:      subsystem, <del> Name:           "automatic_reload_failures_total", Help:           "Total number of failed automatic reloads of encryption configuration split by apiserver identity.", </del> <ins> Name:           "automatic_reloads_total", Help:           "Total number of reload successes and failures of encryption configuration split by apiserver identity.", </ins> StabilityLevel: metrics.ALPHA, }, <ins> []string{"status", "apiserver_id_hash"}, ) // deprecatedEncryptionConfigAutomaticReloadFailureTotal has been deprecated in 1.30.0 // use encryptionConfigAutomaticReloadsTotal instead deprecatedEncryptionConfigAutomaticReloadFailureTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Namespace:         namespace, Subsystem:         subsystem, Name:              "automatic_reload_failures_total", Help:              "Total number of failed automatic reloads of encryption configuration split by apiserver identity.", StabilityLevel:    metrics.ALPHA, DeprecatedVersion: "1.30.0", }, </ins> []string{"apiserver_id_hash"}, ) <del> encryptionConfigAutomaticReloadSuccessTotal = metrics.NewCounterVec( </del> <ins> // deprecatedEncryptionConfigAutomaticReloadSuccessTotal has been deprecated in 1.30.0 // use encryptionConfigAutomaticReloadsTotal instead deprecatedEncryptionConfigAutomaticReloadSuccessTotal = metrics.NewCounterVec( </ins> &metrics.CounterOpts{ <del> Namespace:      namespace, Subsystem:      subsystem, Name:           "automatic_reload_success_total", Help:           "Total number of successful automatic reloads of encryption configuration split by apiserver identity.", StabilityLevel: metrics.ALPHA, </del> <ins> Namespace:         namespace, Subsystem:         subsystem, Name:              "automatic_reload_success_total", Help:              "Total number of successful automatic reloads of encryption configuration split by apiserver identity.", StabilityLevel:    metrics.ALPHA, DeprecatedVersion: "1.30.0", </ins> }, []string{"apiserver_id_hash"}, )

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?


Question: --network "${NETWORK}"  --source-ranges "$2"  --target-tags "$3"  <del> --allow tcp udp icmp esp ah sctp; then </del> <ins> --allow tcp,udp,icmp,esp,ah,sctp; then </ins> if (( attempt > 5 )); then echo -e "${color_red}Failed to create firewall rule $1 ${color_norm}" exit 2

This happens over and over. Scary messages. WARNING: Lists should be separated by commas, try "--allow=tcp,udp,icmp,esp,ah,sctp". If you intend to use udp, icmp, esp, ah, sctp for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--scopes=storage-ro,compute-rw". If you intend to use compute-rw for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--metadata-from-file=startup-script=/usr/local/google/home/thockin/src/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/gce/configure-,kube-env=/tmp/kubernetes.MURmzX/master-kube-". If you intend to use kube-env=/tmp/kubernetes.MURmzX/master-kube- for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--disk=name=e2e-test-thockin-master-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no". If you intend to use device-name=master-pd, mode=rw, boot=no, auto-delete=no for positional arguments, put them before the flag.
That's a new one. gcloud started getting a little more anal here. Exciting. cc
They changed list argument syntax a while back, but the warnings are new. Time to update startup scripts I guess.


Question: } } <ins> func TestPodEvaluatorUsageStats(t *testing.T) { cpu1 := api.ResourceList{api.ResourceCPU: resource.MustParse("1")} tests := []struct { name               string objs               []runtime.Object quotaScopes        []corev1.ResourceQuotaScope quotaScopeSelector *corev1.ScopeSelector want               corev1.ResourceList }{ { name: "nil case", }, { name: "all pods in running state", objs: []runtime.Object{ makePod("p1", "", cpu1, api.PodRunning), makePod("p2", "", cpu1, api.PodRunning), }, want: corev1.ResourceList{ corev1.ResourcePods:               resource.MustParse("2"), corev1.ResourceName("count/pods"): resource.MustParse("2"), corev1.ResourceCPU:                resource.MustParse("2"), corev1.ResourceRequestsCPU:        resource.MustParse("2"), corev1.ResourceLimitsCPU:          resource.MustParse("2"), }, }, { name: "one pods in terminal state", objs: []runtime.Object{ makePod("p1", "", cpu1, api.PodRunning), makePod("p2", "", cpu1, api.PodSucceeded), }, want: corev1.ResourceList{ corev1.ResourcePods:               resource.MustParse("1"), corev1.ResourceName("count/pods"): resource.MustParse("2"), corev1.ResourceCPU:                resource.MustParse("1"), corev1.ResourceRequestsCPU:        resource.MustParse("1"), corev1.ResourceLimitsCPU:          resource.MustParse("1"), }, }, { name: "partial pods matching quotaScopeSelector", objs: []runtime.Object{ makePod("p1", "high-priority", cpu1, api.PodRunning), makePod("p2", "high-priority", cpu1, api.PodSucceeded), makePod("p3", "low-priority", cpu1, api.PodRunning), }, quotaScopeSelector: &corev1.ScopeSelector{ MatchExpressions: []corev1.ScopedResourceSelectorRequirement{ { ScopeName: corev1.ResourceQuotaScopePriorityClass, Operator:  corev1.ScopeSelectorOpIn, Values:    []string{"high-priority"}, }, }, }, want: corev1.ResourceList{ corev1.ResourcePods:               resource.MustParse("1"), corev1.ResourceName("count/pods"): resource.MustParse("2"), corev1.ResourceCPU:                resource.MustParse("1"), corev1.ResourceRequestsCPU:        resource.MustParse("1"), corev1.ResourceLimitsCPU:          resource.MustParse("1"), }, }, { name: "partial pods matching quotaScopeSelector - w/ scopeName specified", objs: []runtime.Object{ makePod("p1", "high-priority", cpu1, api.PodRunning), makePod("p2", "high-priority", cpu1, api.PodSucceeded), makePod("p3", "low-priority", cpu1, api.PodRunning), }, quotaScopes: []corev1.ResourceQuotaScope{ corev1.ResourceQuotaScopePriorityClass, }, quotaScopeSelector: &corev1.ScopeSelector{ MatchExpressions: []corev1.ScopedResourceSelectorRequirement{ { ScopeName: corev1.ResourceQuotaScopePriorityClass, Operator:  corev1.ScopeSelectorOpIn, Values:    []string{"high-priority"}, }, }, }, want: corev1.ResourceList{ corev1.ResourcePods:               resource.MustParse("1"), corev1.ResourceName("count/pods"): resource.MustParse("2"), corev1.ResourceCPU:                resource.MustParse("1"), corev1.ResourceRequestsCPU:        resource.MustParse("1"), corev1.ResourceLimitsCPU:          resource.MustParse("1"), }, }, { name: "partial pods matching quotaScopeSelector - w/ multiple scopeNames specified", objs: []runtime.Object{ makePod("p1", "high-priority", cpu1, api.PodRunning), makePod("p2", "high-priority", cpu1, api.PodSucceeded), makePod("p3", "low-priority", cpu1, api.PodRunning), makePod("p4", "high-priority", nil, api.PodFailed), }, quotaScopes: []corev1.ResourceQuotaScope{ corev1.ResourceQuotaScopePriorityClass, corev1.ResourceQuotaScopeBestEffort, }, quotaScopeSelector: &corev1.ScopeSelector{ MatchExpressions: []corev1.ScopedResourceSelectorRequirement{ { ScopeName: corev1.ResourceQuotaScopePriorityClass, Operator:  corev1.ScopeSelectorOpIn, Values:    []string{"high-priority"}, }, }, }, want: corev1.ResourceList{ corev1.ResourceName("count/pods"): resource.MustParse("1"), }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { gvr := corev1.SchemeGroupVersion.WithResource("pods") listerForPod := map[schema.GroupVersionResource]cache.GenericLister{ gvr: newGenericLister(gvr.GroupResource(), tt.objs), } evaluator := NewPodEvaluator(mockListerForResourceFunc(listerForPod), testingclock.NewFakeClock(time.Now())) usageStatsOption := quota.UsageStatsOptions{ Scopes:        tt.quotaScopes, ScopeSelector: tt.quotaScopeSelector, } actual, err := evaluator.UsageStats(usageStatsOption) if err != nil { t.Error(err) } if !quota.Equals(tt.want, actual.Used) { t.Errorf("expected: %v, actual: %v", tt.want, actual.Used) } }) } } </ins> func TestPodEvaluatorMatchingScopes(t *testing.T) { fakeClock := testingclock.NewFakeClock(time.Now()) evaluator := NewPodEvaluator(nil, fakeClock)

If a Pod is subjected to a with "PriortyClass" as its scope, the quota's calculation is incorrect. See repo steps for more details. with "PriortyClass" as its scope can be calculated at all times. Create a PriorityClass named "p1"Create a quota with as its scope. Note: w/o specifying , this issue won't happenCreate a Job and wait for its completion.Check quota's usage calculation Here is incorrect - it should be . Worse calculation may happen if I create a regular pod and then delete the Job:The eventual result is totally incorrect. No response master, and possibly all versions. kind <details</details<details</details<details</details<details</details
/sig api-machinery /kind bug
/triage accepted
I would like to take this up!
is already in flight for this


Question: testDeletingPodsMetrics(t, 7, metrics.PodGCReasonTerminatingOutOfService) } <ins> func TestGCInspectingPatchedPodBeforeDeletion(t *testing.T) { testCases := []struct { name                 string pod                  *v1.Pod expectedPatchedPod   *v1.Pod expectedDeleteAction *clienttesting.DeleteActionImpl }{ { name: "orphaned pod should have DisruptionTarget condition added before deletion", pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", Name:      "testPod", }, Spec: v1.PodSpec{ NodeName: "deletedNode", }, Status: v1.PodStatus{ Phase: v1.PodRunning, Conditions: []v1.PodCondition{ { Type:   v1.PodReady, Status: v1.ConditionTrue, }, }, }, }, expectedPatchedPod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "default", Name:      "testPod", }, Spec: v1.PodSpec{ NodeName: "deletedNode", }, Status: v1.PodStatus{ Phase: v1.PodFailed, Conditions: []v1.PodCondition{ { Type:    v1.DisruptionTarget, Status:  v1.ConditionTrue, Reason:  "DeletionByPodGC", Message: "PodGC: node no longer exists", }, { Type:   v1.PodReady, Status: v1.ConditionTrue, }, }, }, }, expectedDeleteAction: &clienttesting.DeleteActionImpl{ Name:          "testPod", DeleteOptions: metav1.DeleteOptions{GracePeriodSeconds: pointer.Int64(0)}, }, }, } for _, test := range testCases { t.Run(test.name, func(t *testing.T) { _, ctx := ktesting.NewTestContext(t) defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.PodDisruptionConditions, true)() pods := []*v1.Pod{test.pod} client := setupNewSimpleClient(nil, pods) gcc, podInformer, _ := NewFromClient(ctx, client, -1) gcc.quarantineTime = time.Duration(-1) podInformer.Informer().GetStore().Add(test.pod) gcc.gc(ctx) actions := client.Actions() var patchAction clienttesting.PatchAction var deleteAction clienttesting.DeleteAction for _, action := range actions { if action.GetVerb() == "patch" { patchAction = action.(clienttesting.PatchAction) } if action.GetVerb() == "delete" { deleteAction = action.(clienttesting.DeleteAction) } } if patchAction != nil && test.expectedPatchedPod == nil { t.Fatalf("Pod was pactched but expectedPatchedPod is nil") } if test.expectedPatchedPod != nil { patchedPodBytes := patchAction.GetPatch() originalPod, err := json.Marshal(test.pod) if err != nil { t.Fatalf("Failed to marshal original pod %#v: %v", originalPod, err) } updated, err := strategicpatch.StrategicMergePatch(originalPod, patchedPodBytes, v1.Pod{}) if err != nil { t.Fatalf("Failed to apply strategic merge patch %q on pod %#v: %v", patchedPodBytes, originalPod, err) } updatedPod := &v1.Pod{} if err := json.Unmarshal(updated, updatedPod); err != nil { t.Fatalf("Failed to unmarshal updated pod %q: %v", updated, err) } if diff := cmp.Diff(test.expectedPatchedPod, updatedPod, cmpopts.IgnoreFields(v1.Pod{}, "TypeMeta"), cmpopts.IgnoreFields(v1.PodCondition{}, "LastTransitionTime")); diff != "" { t.Fatalf("Unexpected diff on pod (-want,+got):n%s", diff) } } if deleteAction != nil && test.expectedDeleteAction == nil { t.Fatalf("Pod was deleted but expectedDeleteAction is nil") } if test.expectedDeleteAction != nil { if diff := cmp.Diff(*test.expectedDeleteAction, deleteAction, cmpopts.IgnoreFields(clienttesting.DeleteActionImpl{}, "ActionImpl")); diff != "" { t.Fatalf("Unexpected diff on deleteAction (-want,+got):n%s", diff) } } }) } } </ins> func verifyDeletedAndPatchedPods(t *testing.T, client *fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client)

We should enhance the to verify that adding is unit-testable. We originally thought that this would only be possible with finalizers but there looks to be other options. We could follow examples from In podgc, if we discover an orphaned pod (pod on a missing node), we add a DisruptionCondition to the pod. We only test this behavior in the integration tests because we previously rely on finalizers to halt the deletion so we can avoid flaky tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /sig apps
/good-first-issue /help
Will take a look /assign
/assign
/help
newbie question please, I couldn't find the file in the codebase or the PRs. Where can I read it?
Working on it now!
/assign
/assign
Has done very good work with this so I’m not sure we will be able to accept other PRs. We are waiting for an approval on the PR
is this issue still open
Is the issue still open or awaiting for the approval on the pr?
/remove help /remove good-first-issue
The issue is waiting for approval. I removed the tags to reflect that is is almost done.


Question: }, }, }) <ins> if apierrors.IsForbidden(err) { // The user is not allowed to create ClusterRoleBindings. This // probably means that RBAC is not being used. If RBAC is being // used, this test will probably fail later. framework.Logf("Attempt to create ClusterRoleBinding was forbidden: %v.", err) return } </ins> framework.ExpectNoError(err) err = framework.WaitForAuthorizationUpdate(jig.client.Authorization(),

Three tests now create RoleBindings to give permissions to test service accounts: NodeProblemDetector Ingress PreStop Creating RBAC Rolebindings currently requires the user issuing the create call to have the required permissions. If RBAC is not enabled, and the cluster hasn't been configured with any RBAC bootstrapping, that won't be possible. We should make this test code conditional on whether RBAC is in use.
c.f.
Quick fix is to tolerate forbidden errors there and assume e2e setups using methods other than RBAC have already granted permissions to the service accounts
I'll put together a PR to tolerate the 403 in the test helper.
I think this can be closed now that the e2e RBAC setup is best effort and failures there don't affect the test if another authorizer is in use
I agree. My original thought was that we'd make the RBAC test stuff only happen on RBAC-enabled setups. But, since we're going to be pushing setups towards using RBAC, we might as well have those pieces stick around in the common code (as best-effort).


Question: configs[ResourceConsumer] = Config{list.PromoterE2eRegistry, "resource-consumer", "1.13"} configs[SdDummyExporter] = Config{list.GcRegistry, "sd-dummy-exporter", "v0.2.0"} configs[VolumeNFSServer] = Config{list.PromoterE2eRegistry, "volume/nfs", "1.3"} <del> configs[VolumeISCSIServer] = Config{list.PromoterE2eRegistry, "volume/iscsi", "2.5"} </del> <ins> configs[VolumeISCSIServer] = Config{list.PromoterE2eRegistry, "volume/iscsi", "2.6"} </ins> configs[VolumeRBDServer] = Config{list.PromoterE2eRegistry, "volume/rbd", "1.0.6"} configs[WindowsServer] = Config{list.MicrosoftRegistry, "windows", "1809"}

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question:
message

String detailing the error encountered during Attach or Detach operation. This string maybe logged, so it should not contain sensitive information.

String detailing the error encountered during Attach or Detach operation. This string may be logged, so it should not contain sensitive information.

false

string

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package reporters import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" "strings" "time" "k8s.io/klog" "github.com/onsi/ginkgo/config" "github.com/onsi/ginkgo/types" ) // ProgressReporter is a ginkgo reporter which tracks the total number of tests to be run/passed/failed/skipped. // As new tests are completed it updates the values and prints them to stdout and optionally, sends the updates // to the configured URL. type ProgressReporter struct { LastMsg string `json:"msg"` TestsTotal     int `json:"total"` TestsCompleted int `json:"completed"` TestsSkipped   int `json:"skipped"` TestsFailed    int `json:"failed"` Failures []string `json:"failures,omitempty"` progressURL string client      *http.Client } // NewProgressReporter returns a progress reporter which posts updates to the given URL. func NewProgressReporter(progressReportURL string) *ProgressReporter { rep := &ProgressReporter{ Failures:    []string{}, progressURL: progressReportURL, } if len(progressReportURL) > 0 { rep.client = &http.Client{ Timeout: time.Second * 10, } } return rep } // SpecSuiteWillBegin is invoked by ginkgo when the suite is about to start and is the first point in which we can // antipate the number of tests which will be run. func (reporter *ProgressReporter) SpecSuiteWillBegin(cfg config.GinkgoConfigType, summary *types.SuiteSummary) { reporter.TestsTotal = summary.NumberOfSpecsThatWillBeRun reporter.LastMsg = "Test Suite starting" reporter.sendUpdates() } // SpecSuiteDidEnd is the last method invoked by Ginkgo after all the specs are run. func (reporter *ProgressReporter) SpecSuiteDidEnd(summary *types.SuiteSummary) { reporter.LastMsg = "Test Suite completed" reporter.sendUpdates() } // SpecDidComplete is invoked by Ginkgo each time a spec is completed (including skipped specs). func (reporter *ProgressReporter) SpecDidComplete(specSummary *types.SpecSummary) { testname := strings.Join(specSummary.ComponentTexts[1:], " ") switch specSummary.State { case types.SpecStateFailed: if len(specSummary.ComponentTexts) > 0 { reporter.Failures = append(reporter.Failures, testname) } else { reporter.Failures = append(reporter.Failures, "Unknown test name") } reporter.TestsFailed++ reporter.LastMsg = fmt.Sprintf("FAILED %v", testname) case types.SpecStatePassed: reporter.TestsCompleted++ reporter.LastMsg = fmt.Sprintf("PASSED %v", testname) case types.SpecStateSkipped: reporter.TestsSkipped++ return default: return } reporter.sendUpdates() } // sendUpdates serializes the current progress and prints it to stdout and also posts it to the configured endpoint if set. func (reporter *ProgressReporter) sendUpdates() { b := reporter.serialize() fmt.Println(string(b)) go reporter.postProgressToURL(b) } func (reporter *ProgressReporter) postProgressToURL(b []byte) { // If a progressURL and client is set/available then POST to it. Noop otherwise. if reporter.client == nil || len(reporter.progressURL) == 0 { return } resp, err := reporter.client.Post(reporter.progressURL, "application/json", bytes.NewReader(b)) if err != nil { klog.Errorf("Failed to post progress update to %v: %v", reporter.progressURL, err) return } if resp.StatusCode >= 400 { klog.Errorf("Unexpected response when posting progress update to %v: %v", reporter.progressURL, resp.StatusCode) if resp.Body != nil { defer resp.Body.Close() respBody, err := ioutil.ReadAll(resp.Body) if err != nil { klog.Errorf("Failed to read response body from posting progress: %v", err) return } klog.Errorf("Response body from posting progress update: %v", respBody) } return } } func (reporter *ProgressReporter) serialize() []byte { b, err := json.Marshal(reporter) if err != nil { return []byte(fmt.Sprintf(`{"msg":"%v", "error":"%v"}`, reporter.LastMsg, err)) } return b } // SpecWillRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) SpecWillRun(specSummary *types.SpecSummary) {} // BeforeSuiteDidRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) BeforeSuiteDidRun(setupSummary *types.SetupSummary) {} // AfterSuiteDidRun is implemented as a noop to satisfy the reporter interface for ginkgo. func (reporter *ProgressReporter) AfterSuiteDidRun(setupSummary *types.SetupSummary) {} </ins>

What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale