Question: import ( "errors" "fmt" <ins> "io/ioutil" </ins> "os" "path/filepath" "strings"

I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:e45d35f9f967ef7d49496d1d2e6f9ad378a29012b0c90f78b5d59a254f3135e6", </del> <ins> digest = "sha256:f3a37c4d8700a5ff454d94a2bef7d165d287759cea737a621c20e4aa3891dbbb", </ins> registry = "gcr.io", repository = "google-containers/debian-hyperkube-base-amd64", <del> tag = "0.3",  # ignored, but kept here for documentation </del> <ins> tag = "0.4",  # ignored, but kept here for documentation </ins> ) docker_pull(

Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I update the kubelet on a node from 1.7.x to 1.8.0, pods on that node stop being able to connect to services whose pods happen to be on the same node. I use the Kubenet network plugin. What you expected to happen: Pods should be able to connect to a service even if the services's pods are on the same node. How to reproduce it (as minimally and precisely as possible): a cluster the DNS addon another pod that runs on the same node as one of the DNS pods resolution starts to fail when the service redirects the requests to the pod that is on the same node. Anything else we need to know?: I have some packet captures that I can show. Kubelet: 1.7.5 DNS pod: 10.2.4.76 Origin pod: 10.2.4.64 Kubelet: 1.8.0 DNS pod: 10.2.1.254 Origin pod: 10.2.1.234 We can see that in the requests that fail the DNS IP is correctly DNATed to the DNS POD destination address but the response is not masqueraded, it still carries the DNS POD source (it should have an origin of 10.3.0.10). Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure StandardA4 OS (e.g. from /etc/os-release): "Container Linux by CoreOS 1465.8.0 (Ladybug)" Kernel (e.g. ): Linux node-1-vm 4.12.14-coreos SMP Wed Sep 20 22:20:05 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
/sig node
The problem appears when transitioning from kubelet v1.8.0-alpha.2 to v1.8.0-alpha.3.
Well, I've found the problem. Starting from v1.8.0-alpha.3 the "modprobe" binary is not included in the image and so the br-netfilter kernel module is not loaded. I'm using the image, although through "kubelet-wrapper" from CoreOS. Where should this binary reside?

cc/ 1.8 release team.
There is a PR from so it doesn't get lost.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: if ! which salt-call >/dev/null 2>&1; then # Install salt binaries curl -sS -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s <ins> # Fedora >= 23 includes salt packages but the bootstrap is # creating configuration for a (non-existent) salt repo anyway. # Remove the invalid repo to prevent dnf from warning about it on # every update.  Assume this problem is specific to Fedora 23 and # will fixed by the time another version of Fedora lands. local fedora_version=$(grep 'VERSION_ID' /etc/os-release | sed 's+VERSION_ID=++') if [[ "${fedora_version}" = '23' ]]; then local repo_file='/etc/yum.repos.d/saltstack-salt-fedora-23.repo' if [[ -f "${repo_file}" ]]; then rm "${repo_file}" fi fi </ins> fi }

The latest vagrant always restarts nfs when nfs is used to share folders: This doesn't work with the default kube libvirt image: Attempting to restart nfs (systemctl restart nfs) on this image results in a failure from /usr/sbin/exportfs (invoked in the nfs unit file via ExecStartPre) of 'No file systems exported!' and this halts vagrant provisioning. To replicate (centos 7.1+vagrant 1.8.1+vagrant-libvirt 0.0.32):
This issue should be resolved by (already merged) and .
I marked as lgtm, so its basically blocked by the submit queue being fixed.


Question: go_test( name = "go_default_test", <del> srcs = [ "compatibility_test.go", "defaults_test.go", ], </del> <ins> srcs = ["defaults_test.go"], </ins> embed = [":go_default_library"], deps = [ <del> "//pkg/apis/core/install:go_default_library", </del> "//pkg/scheduler/algorithm/predicates:go_default_library", <del> "//pkg/scheduler/api:go_default_library", "//pkg/scheduler/api/latest:go_default_library", "//pkg/scheduler/factory:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//staging/src/k8s.io/client-go/util/testing:go_default_library", </del> ], )

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: If you contributed in phase 1, please give others a chance before claiming this issue. You can also help out by reviewing the phase 2 PRs. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /sig scheduling /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to take it and thanks
Hi! I am contributer and I would like to take it.
while you're working on do you mind letting take a crack at this one?
OK, go head
, thanks!
/lifecycle active for
Thanks for taking care of this one! :smiley:
You are wellcome


Question: } const ( <del> proxyModeUserspace              = "userspace" proxyModeIPTables               = "iptables" experimentalProxyModeAnnotation = options.ExperimentalProxyModeAnnotation betaProxyModeAnnotation         = "net.beta.kubernetes.io/proxy-mode" </del> <ins> proxyModeUserspace = "userspace" proxyModeIPTables  = "iptables" </ins> ) func checkKnownProxyMode(proxyMode string) bool {

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: var zeroIPv4 = net.ParseIP("0.0.0.0") var localhostIPv4 = net.ParseIP("127.0.0.1") <del> var zeroIPv6 = net.ParseIP("::0") </del> <ins> var zeroIPv6 = net.ParseIP("::") </ins> var localhostIPv6 = net.ParseIP("::1") // Build a slice of iptables args that are common to from-container and from-host portal rules.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: ) predicateResults := make(map[string]HostPredicate) <del> if ecache != nil { // getHashEquivalencePod will return immediately if no equivalence pod found equivalenceHash, eCacheAvailable = ecache.getHashEquivalencePod(pod) } </del> podsAdded := false // We run predicates twice in some cases. If the node has greater or equal priority // nominated pods, we run them when those pods are added to meta and nodeInfo.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: kubeNodePortLocalSetUDPComment = "Kubernetes nodeport UDP port with externalTrafficPolicy=local" kubeNodePortLocalSetUDP        = "KUBE-NODE-PORT-LOCAL-UDP" <del> kubeNodePortSetSCTPComment = "Kubernetes nodeport SCTP port for masquerade purpose" kubeNodePortSetSCTP        = "KUBE-NODE-PORT-SCTP" </del> <ins> // This ipset is no longer active but still used in previous versions. // DO NOT create an ipset using this name legacyKubeNodePortSetSCTPComment = "Kubernetes nodeport SCTP port for masquerade purpose" legacyKubeNodePortSetSCTP        = "KUBE-NODE-PORT-SCTP" </ins> <del> kubeNodePortLocalSetSCTPComment = "Kubernetes nodeport SCTP port with externalTrafficPolicy=local" kubeNodePortLocalSetSCTP        = "KUBE-NODE-PORT-LOCAL-SCTP" </del> <ins> // This ipset is no longer active but still used in previous versions. // DO NOT create an ipset using this name legacyKubeNodePortLocalSetSCTPComment = "Kubernetes nodeport SCTP port with externalTrafficPolicy=local" legacyKubeNodePortLocalSetSCTP        = "KUBE-NODE-PORT-LOCAL-SCTP" kubeNodePortSetSCTPComment = "Kubernetes nodeport SCTP port for masquerade purpose with type 'hash ip:port'" kubeNodePortSetSCTP        = "KUBE-NODE-PORT-SCTP-HASH" kubeNodePortLocalSetSCTPComment = "Kubernetes nodeport SCTP port with externalTrafficPolicy=local with type 'hash ip:port'" kubeNodePortLocalSetSCTP        = "KUBE-NODE-PORT-LOCAL-SCTP-HASH" </ins> ) // IPSetVersioner can query the current ipset version.

What happened: When updating to 1.15 in IPVS mode, kube-proxy fails to start What you expected to happen: kube-proxy updates ok How to reproduce it (as minimally and precisely as possible): Deploy kube-proxy <1.15 in ipvs mode and update to 1.15 Anything else we need to know?: Error message in kube-proxy: I think the reason is the change of ipset type (bitmap to hash) in PR The 2 sets that creates the problem are: - Removing the sets manually works but is not a great solution for in-place updates. I think we should fix this before the 1.15 release (not sure what the best approach is: delete the set if it exists with the wrong type? Use a different name to avoid the collision?) Environment: Kubernetes version (use ): 1.14 =1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): ubuntu 18.04 Kernel (e.g. ): 4.4 /sig network /area ipvs /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by
Great catch! Looks like a breaking change. One solution like you mentioned is to add support for replacing ipsets in if a similar ipset already exists? Thoughts
I noticed we pass in for ipsets but that only checks for identical ipset types. Maybe needs to be smarter and do a full diff
This could work as well
Opened a fix for it here , let's chat there :)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


Question: // If certain nodes were deleted after the last snapshot was taken, we should remove them from the snapshot. func (cache *schedulerCache) removeDeletedNodesFromSnapshot(snapshot *Snapshot) { <del> toDelete := len(snapshot.nodeInfoMap) - len(cache.nodes) </del> <ins> toDelete := len(snapshot.nodeInfoMap) - cache.nodeTree.numNodes </ins> for name := range snapshot.nodeInfoMap { if toDelete <= 0 { break } <del> if _, ok := cache.nodes[name]; !ok { </del> <ins> if n, ok := cache.nodes[name]; !ok || n.info.Node() == nil { </ins> delete(snapshot.nodeInfoMap, name) toDelete-- }

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: uniqueRS := rs.String() if _, ok := q.list[uniqueRS]; ok { <del> return false </del> <ins> delete(q.list, uniqueRS) return true </ins> } <del> delete(q.list, uniqueRS) return true </del> <ins> return false </ins> } func (q *graceTerminateRSList) flushList(handler func(rsToDelete *listItem) (bool, error)) bool {

What happened: On pods with a lot of DNS traffic, many DNS queries to the DNS ClusterIP fail during a rolling update of the DNS pods (in our case, we use coredns). What you expected to happen: Almost all DNS queries should succeed during a rolling update. How to reproduce it (as minimally and precisely as possible): Create a coredns (this would probably be the same with kube-dns) service backed by 2 pods Access this service by creating a lot of DNS queries, for example: Wait for the IPVS connection table to contain a lot of DNS connections (a few minutes) Delete a DNS pod A significant number of queries will timeout. After 5mn (depending in the IPVS timeout configuration) everything will work ok again. Anything else we need to know?: This comes from the current logic in the graceful termination code: The current logic is to delete a RS if the number of active connections is 0. This makes sense for TCP but for UDP the number of active connections is always 0. This is an issue for DNS queries because the RS will be deleted but the IPVS connection will remain until it expires (5mn by default) and if there are a lot of DNS queries, the port will be reused and queries blackholed. In addition, will not always work here because IPVS only removes connections when a packet is processed (in the case of DNS once the client receives an answer, no more packets are sent). Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign
I think there is also an issue with the way we clear the conntrack for UDP We call in : If I understand correctly this will clear conntrack entries matching and before graceful termination is done. I think we should only remove the conntrack entries when we delete the real server. In addition, we only clear the conntrack entries for <clusterIP, I think we should also clean them for Node IPs (Nodeports) and External IPs. What do you think?
I think we still need to keep this issue open even though we have merged - I assume they are different issues.

yes I think the PR is a great improvement but we probably need a few other small fixes (like the conntrack thing I mentioned)
The fix from the PR is not perfect yet for DNS traffic when qps is high (but it's definitely an improvement for UDP traffic which didn't support any kind of graceful termination). For example: Will generate 5000qps and ~500 new entries in the IPVS table per second. With a UDP timeout of 30s this leads to entries maximum. If we have 2 pods, when we delete one we have to wait for the ~7500 entries to expire, given the timeout (30s) many ports will be reused (which resets the timeout) and when the coredns pod finishes its lameduck period, we lose queries. 5000 qps against 2 coredns pods is high and unlikely but to decrease the probability of DNS queries failing, we can either: decrease the UDP timeout (in this example if set to 10s, we dont lose a single packet) increase the lameduck period (I tested with UDP timeout 30s and lameduck 180s and it worked great) (of course with more pods the impact is less important: what matters is the number of ports assigned to Terminating pods) I think being able to control the UDP timeout is very important, the default (300s) is very high (see: ) (I forked dnstrace to add the parameter: , I'll try and get it merged in the main repository)
can you please help confirm our issue? in our system, our dns query is quite often, like 1 query per 10 seconds. in this case, we found that if we restart coredns, the old IPVS rules are never deleted on some of our nodes, which caused a lot of failed dns queries. is this because the graceful termination mechanism introduced here? In order to delete the RS and remove the ipvs rules, we have to wait the DNS query expire, but before the old dns query expires, new dns query came in, so grace termination will never see and old ipvs rules are never cleaned up, causing dns query issue.

if this is the case, I would suggest to introduce a force termination mechanism
: it depends how you query DNS: if you have a persistent resolution socket, the IPVS backend may never expire but in that case your resolver should timeout and reconnect which will use an active backend. If you use a standard resolver (glibc for instance) each query will get a new connection so you will reach an active coredns unless you query frequently enough so you end up reusing a same source port associated with a removed backend (but below 100 queries/second this is very unlikely). I have ideas to address both use cases: configure more aggressive timeouts (in particular for UDP) and make them configurable (see issue: ) forcefully remove real servers (coredns in your case) that are still tracked by IPVS after X minutes (today we only delete them when they have 0 active and 0 inactive connections which may never happen depending on your workload) may have ideas on this topic too
DNS resolution through kube-proxy is tricky for several reasons regardless of the proxy mode (for iptables for instance, many people have conntrack issues when doing a lot of queries). In our case, we are experimenting with node local coredns caches (you can also have a look at which is an implementation of this idea)
isn't 100 queries/second quite common for a site? QPS can be thousands nowadays. we did not do special configuration for our dns so it should be using glibc. I think what you mentioned about reusing the same source port should be our issue here. by checking the output of ipvsadmin, it seems the source port is reused for different udp queries <img width="517" alt="" src="https://user-"do you have any other suggestions on this issue?
btw, forcefully remove real servers is what I want here.
I meant 100 qps for a single client (which can definitely happen though) Yes it is very likely that you have the issue I mentioned Until we implement the solution I mentioned natively, you can try changing the ipvs udp timeout on your instances. The default are: You can change udp timeout to 30s (or even lower depending on your connection rate): (0 here means do not touch tcp/tcpfin)


Question: return metrics.NewKubeRegistry() } <del> func AssertVectorCount(t *testing.T, name string, labelFilter map[string]string, wantCount int) { </del> <ins> func AssertVectorCount(t TB, name string, labelFilter map[string]string, wantCount int) { </ins> metrics, err := legacyregistry.DefaultGatherer.Gather() if err != nil { t.Fatalf("Failed to gather metrics: %s", err)

We have many "test only" packages through out the project, containing fixtures, helpers, utilities, etc... These test libraries are not intended for use in production binaries. They often lack rigorous review and sufficient testing for use in production binaries. We should make sure they don't accidentally get pulled in. Strawman proposal, test only packages must import "testing". Blank import is sufficient: Then write a presubmit that fails if we find symbol in the release binary. This approach is trivial and fairly effective although a more sophisticated approach could print a dep chain.
/assign
I hope I will be able to contribute once this gets marked as .
/triage accepted
I think that will be enough a new script or a check in the building scripts /cc since he has experience from the providerless building effort
I got what we want to achieve here but I'm little confused about: If I execute: after adding the import then only it is giving - what do we want to achieve by executing this command exactly? I got what it is doing. I've opened a thread in slack as well:
Hello! Is there anyone working on this issue at the moment? If not and the issue it's still relevant, I would like to work on it if possible. Thank you.
I opened a PR in the meantime, if it's not needed anymore, I can close it.


Question: t.Errorf("unexpected non-error") } } <ins> func TestNoSelectorUnknowResourceType(t *testing.T) { b := newDefaultBuilder(). NamespaceParam("test"). ResourceTypeOrNameArgs(false, "unknown") err := b.Do().Err() if err != nil { if !strings.Contains(err.Error(), "server doesn't have a resource type "unknown"") { t.Fatalf("unexpected error: %v", err) } } } </ins> func TestSingleResourceType(t *testing.T) { b := newDefaultBuilder(). LabelSelectorParam("a=b").

Hello! I got this error: or:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do I need to use ?
use instead


Question: // Can't get the network tier, just return an error. return "", err } <ins> // containsCIDR returns true if outer contains inner. func containsCIDR(outer, inner *net.IPNet) bool { return outer.Contains(firstIPInRange(inner)) && outer.Contains(lastIPInRange(inner)) } // firstIPInRange returns the first IP in a given IP range. func firstIPInRange(ipNet *net.IPNet) net.IP { return ipNet.IP.Mask(ipNet.Mask) } // lastIPInRange returns the last IP in a given IP range. func lastIPInRange(cidr *net.IPNet) net.IP { ip := append([]byte{}, cidr.IP...) for i, b := range cidr.Mask { ip[i] |= ^b } return ip } // subnetsInCIDR takes a list of subnets for a single region and // returns subnets which exists in the specified CIDR range. func subnetsInCIDR(subnets []*compute.Subnetwork, cidr *net.IPNet) ([]*compute.Subnetwork, error) { var res []*compute.Subnetwork for _, subnet := range subnets { _, subnetRange, err := net.ParseCIDR(subnet.IpCidrRange) if err != nil { return nil, fmt.Errorf("unable to parse CIDR %q for subnet %q: %v", subnet.IpCidrRange, subnet.Name, err) } if containsCIDR(cidr, subnetRange) { res = append(res, subnet) } } return res, nil } type netType string const ( netTypeLegacy netType = "LEGACY" netTypeAuto   netType = "AUTO" netTypeCustom netType = "CUSTOM" ) func typeOfNetwork(network *compute.Network) netType { if network.IPv4Range != "" { return netTypeLegacy } if network.AutoCreateSubnetworks { return netTypeAuto } return netTypeCustom } </ins>

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: apiVersion: apps/v1 kind: Deployment metadata: <del> name: metrics-server-v0.5.0 </del> <ins> name: metrics-server-v0.5.1 </ins> namespace: kube-system labels: k8s-app: metrics-server addonmanager.kubernetes.io/mode: Reconcile <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> spec: selector: matchLabels: k8s-app: metrics-server <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> template: metadata: name: metrics-server labels: k8s-app: metrics-server <del> version: v0.5.0 </del> <ins> version: v0.5.1 </ins> spec: securityContext: seccompProfile:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As described in issue , we found kubernetes have been running normally, but the control-plane nodes's metrics value of startTime is similar to the timestamp. Get the correct metrics of node As described in issue anything else we need to provide? As described in issue Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/remove-sig node /sig instrumentation /assign
I think the issue is about Summary API having start time for node metric, but this timestamp is incorrect. This is node issue, not instrumentation. /sig node /remove-sig instrumentation /unassign
When discussing adding node start time metric to /metrics/resource endpoint I remember that SIG Node response to this was that there is no good way to define node start time. I don't think we will be able to provide correct value here. As summary API is technically deprecated, it makes no sense to fix it. We should warn users against using it and wait for removal of Summary API.
Okay, let's wait for the release of metrics-server v0.6.0
Do you know if this is fixed in metrics server?
/triage needs-information Can you please add info on name of timeserias and other steps to repro ti understand exactly what is needed here. Thank you!

in metrics server v0.6.0, we will use /metrics/resource Kubelet endpoint instead summary API
as issue described , the result is: for example : the startup time of the node test-us-west-1-control-plane-577lp is "startTime": "2021-08-03T19:33:50Z",, and timestamp reported by metrics is "2021-08-03T19:34:36Z", but as described, output from control plane node


Question: deleteRSTime := time.Now().Sub(startTime) Logf("Deleting RS %s took: %v", name, deleteRSTime) if err == nil { <del> err = waitForReplicaSetPodsGone(clientset, rc) </del> <ins> err = waitForReplicaSetPodsGone(clientset, rs) </ins> } terminatePodTime := time.Now().Sub(startTime) - deleteRSTime Logf("Terminating ReplicaSet %s pods took: %v", name, terminatePodTime)

As the title said, we need update touse server side reaper. xref:
/sig apps
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign


Question: find-release-tars upload-server-tars elif [[ ${KUBE_VERSION} =~ ${KUBE_VERSION_REGEX} ]]; then <del> KUBE_TAR_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes.tar.gz" </del> SERVER_BINARY_TAR_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes-server-linux-amd64.tar.gz" SALT_TAR_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes-salt.tar.gz" elif [[ ${KUBE_VERSION} =~ ${KUBE_CI_VERSION_REGEX} ]]; then <del> KUBE_TAR_URL="https://storage.googleapis.com/kubernetes-release/ci/${KUBE_VERSION}/kubernetes.tar.gz" </del> SERVER_BINARY_TAR_URL="https://storage.googleapis.com/kubernetes-release/ci/${KUBE_VERSION}/kubernetes-server-linux-amd64.tar.gz" SALT_TAR_URL="https://storage.googleapis.com/kubernetes-release/ci/${KUBE_VERSION}/kubernetes-salt.tar.gz" else echo "Version doesn't match regexp" >&2 exit 1 fi <del> until KUBE_TAR_HASH=$(curl --fail --silent "${KUBE_TAR_URL}.sha1"); do echo "Failure trying to curl release .sha1" done until SERVER_BINARY_TAR_HASH=$(curl --fail --silent "${SERVER_BINARY_TAR_URL}.sha1"); do </del> <ins> if ! SERVER_BINARY_TAR_HASH=$(curl -Ss --fail "${SERVER_BINARY_TAR_URL}.sha1"); then </ins> echo "Failure trying to curl release .sha1" <del> done until SALT_TAR_HASH=$(curl --fail --silent "${SALT_TAR_URL}.sha1"); do </del> <ins> fi if ! SALT_TAR_HASH=$(curl -Ss --fail "${SALT_TAR_URL}.sha1"); then </ins> echo "Failure trying to curl Salt tar .sha1" <del> done if ! curl -Ss --range 0-1 "${KUBE_TAR_URL}" >&/dev/null; then echo "Can't find release at ${KUBE_TAR_URL}" >&2 exit 1 </del> fi <del> if ! curl -Ss --range 0-1 "${SERVER_BINARY_TAR_URL}" >&/dev/null; then </del> <ins> if ! curl -Ss --head "${SERVER_BINARY_TAR_URL}" >&/dev/null; then </ins> echo "Can't find release at ${SERVER_BINARY_TAR_URL}" >&2 exit 1 fi <del> if ! curl -Ss --range 0-1 "${SALT_TAR_URL}" >&/dev/null; then </del> <ins> if ! curl -Ss --head "${SALT_TAR_URL}" >&/dev/null; then </ins> echo "Can't find Salt tar at ${SALT_TAR_URL}" >&2 exit 1 fi } <ins> # Verify and find the various tar files that we are going to use on the server. # # Assumed vars: #   KUBE_ROOT # Vars set: #   SERVER_BINARY_TAR #   SALT_TAR function find-release-tars() { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" >&2 exit 1 fi SALT_TAR="${KUBE_ROOT}/server/kubernetes-salt.tar.gz" if [[ ! -f "$SALT_TAR" ]]; then SALT_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-salt.tar.gz" fi if [[ ! -f "$SALT_TAR" ]]; then echo "!!! Cannot find kubernetes-salt.tar.gz" >&2 exit 1 fi } </ins>

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: fi fi <del> KUBE_ROOT_HASH=$(kube::build::short_hash "${HOSTNAME:-}:${KUBE_ROOT}") </del> <ins> KUBE_GIT_BRANCH=$(git symbolic-ref --short -q HEAD 2>/dev/null || true) KUBE_ROOT_HASH=$(kube::build::short_hash "${HOSTNAME:-}:${KUBE_ROOT}:${KUBE_GIT_BRANCH}") </ins> KUBE_BUILD_IMAGE_TAG_BASE="build-${KUBE_ROOT_HASH}" KUBE_BUILD_IMAGE_TAG="${KUBE_BUILD_IMAGE_TAG_BASE}-${KUBE_BUILD_IMAGE_VERSION}" KUBE_BUILD_IMAGE="${KUBE_BUILD_IMAGE_REPO}:${KUBE_BUILD_IMAGE_TAG}"

When I change branches, I really want the prior build invalidated. Could we consider the git branch (if this is a git repo) in the short_hash that uniquifies the build?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig testing ?
/remove-sig testing
/sig release (on the presumption that build lands there)


Question: return instances.InstanceExistsByProviderID(ctx, providerID) } <del> func getNodeAddressesByProviderIDOrName(ctx context.Context, instances cloudprovider.Instances, node *v1.Node) ([]v1.NodeAddress, error) { nodeAddresses, err := instances.NodeAddressesByProviderID(ctx, node.Spec.ProviderID) </del> <ins> func getNodeAddressesByProviderIDOrName(ctx context.Context, instances cloudprovider.Instances, providerID, nodeName string) ([]v1.NodeAddress, error) { nodeAddresses, err := instances.NodeAddressesByProviderID(ctx, providerID) </ins> if err != nil { providerIDErr := err <del> nodeAddresses, err = instances.NodeAddresses(ctx, types.NodeName(node.Name)) </del> <ins> nodeAddresses, err = instances.NodeAddresses(ctx, types.NodeName(nodeName)) </ins> if err != nil { return nil, fmt.Errorf("error fetching node by provider ID: %v, and error by node name: %v", providerIDErr, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re:


Question: func AddFlags(options *Options, fs *pflag.FlagSet) { fs.StringVar(&options.ConfigFile, "config", options.ConfigFile, "The path to the configuration file.") fs.StringVar(&options.WriteConfigTo, "write-config-to", options.WriteConfigTo, "If set, write the default configuration values to this file and exit.") <del> fs.BoolVar(&options.CleanupAndExit, "cleanup-iptables", options.CleanupAndExit, "If true cleanup iptables rules and exit.") </del> <ins> fs.BoolVar(&options.CleanupAndExit, "cleanup-iptables", options.CleanupAndExit, "If true, cleanup iptables rules and exit.") </ins> // All flags below here are deprecated and will eventually be removed.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug What happened: Got panic after running: Panic: What you expected to happen: No panic and exit successfully. How to reproduce it (as minimally and precisely as possible): Run the following command on both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Happened with the following versions: v1.7.0-alpha.3 v1.7.0-alpha.4 v1.7.0-beta.0 v1.7.0-beta.1 v1.7.0-beta.2 v1.7.0-rc.1 v1.8.0-alpha.0 v1.8.0-alpha.1 Anything else we need to know?: I guess it caused by . Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Kernel (e.g. ): CoreOS v1298.7.0: Linux node- 4.9.16-coreos-r1 SMP Tue Apr 4 19:09:38 UTC 2017 x8664 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz GenuineIntel GNU/Linux Ubuntu 16.04.2 LTS: Linux ubuntu- 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network /area kube-proxy
/cc
cc/ Looks like there is a new regression in v1.7. Please triage this to decide if it is a release blocker. If not, please provide the workaround for the users. Thanks!

cc/
Digging into the code, will send a PR.
PR sent. Folks review it please.
Added the release-blocker label. We got a PR for it very fast so it won't delay the release anyway, thanks


Question: cpus := m.state.GetCPUSetOrDefault(containerID) m.Unlock() <del> err = m.updateContainerCPUSet(containerID, cpus) if err != nil { glog.Errorf("[cpumanager] AddContainer error: %v", err) return err </del> <ins> if !cpus.IsEmpty() { err = m.updateContainerCPUSet(containerID, cpus) if err != nil { glog.Errorf("[cpumanager] AddContainer error: %v", err) return err } } else { glog.V(5).Infof("[cpumanager] update container resources is skipped due to cpu set is empty") </ins> } return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There will always be extra UpdateContainerResources CRI call even when cpu set is empty. This is unneeded. What you expected to happen: Only call UpdateContainerResources when cpu set is set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: } type pluginTestConfig struct { <del> medium                        v1.StorageMedium idempotent                    bool </del> <ins> medium v1.StorageMedium //volumeDirExists indicates whether volumeDir already/still exists before volume setup/teardown volumeDirExists bool //readyDirExists indicates whether readyDir already/still exists before volume setup/teardown readyDirExists                bool </ins> expectedSetupMounts           int shouldBeMountedBeforeTeardown bool expectedTeardownMounts        int

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package podtopologyspread import ( "context" "testing" "k8s.io/api/core/v1" "k8s.io/kubernetes/pkg/scheduler/algorithm/predicates" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/migration" "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" schedulernodeinfo "k8s.io/kubernetes/pkg/scheduler/nodeinfo" st "k8s.io/kubernetes/pkg/scheduler/testing" ) var hardSpread = v1.DoNotSchedule func TestPodTopologySpreadFilter_SingleConstraint(t *testing.T) { tests := []struct { name         string pod          *v1.Pod nodes        []*v1.Node existingPods []*v1.Pod fits         map[string]bool }{ { name: "no existing pods", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, "node-x": true, "node-y": true, }, }, { name: "no existing pods, incoming pod doesn't match itself", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("bar").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, "node-x": true, "node-y": true, }, }, { name: "existing pods with mis-matched namespace doens't count", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Namespace("ns1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Namespace("ns2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-x1").Node("node-x").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, "node-x": false, "node-y": false, }, }, { name: "pods spread across zones as 3/3, all nodes fit", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, "node-x": true, "node-y": true, }, }, { // TODO(Huang-Wei): maybe document this to remind users that typos on node labels // can cause unexpected behavior name: "pods spread across zones as 1/2 due to absence of label 'zone' on node-b", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zon", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-x1").Node("node-x").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": false, "node-x": false, "node-y": false, }, }, { name: "pods spread across nodes as 2/1/0/3, only node-x fits", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": false, "node-x": true, "node-y": false, }, }, { name: "pods spread across nodes as 2/1/0/3, maxSkew is 2, node-b and node-x fit", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 2, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": true, "node-x": true, "node-y": false, }, }, { // not a desired case, but it can happen // TODO(Huang-Wei): document this "pod-not-match-itself" case // in this case, placement of the new pod doesn't change pod distribution of the cluster // as the incoming pod doesn't have label "foo" name: "pods spread across nodes as 2/1/0/3, but pod doesn't match itself", pod: st.MakePod().Name("p").Label("bar", "").SpreadConstraint( 1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": true, "node-x": true, "node-y": false, }, }, { // only node-a and node-y are considered, so pods spread as 2/~1~/~0~/3 // ps: '~num~' is a markdown symbol to denote a crossline through 'num' // but in this unit test, we don't run NodeAffinityPredicate, so node-b and node-x are // still expected to be fits; // the fact that node-a fits can prove the underlying logic works name: "incoming pod has nodeAffinity, pods spread as 2/~1~/~0~/3, hence node-a fits", pod: st.MakePod().Name("p").Label("foo", ""). NodeAffinityIn("node", []string{"node-a", "node-y"}). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, // in real case, it's false "node-x": true, // in real case, it's false "node-y": false, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { nodeInfoMap := schedulernodeinfo.CreateNodeNameToInfoMap(tt.existingPods, tt.nodes) meta := predicates.GetPredicateMetadata(tt.pod, nodeInfoMap) state := v1alpha1.NewCycleState() state.Write(migration.PredicatesStateKey, &migration.PredicatesStateData{Reference: meta}) plugin, _ := New(nil, nil) for _, node := range tt.nodes { status := plugin.(*PodTopologySpread).Filter(context.Background(), state, tt.pod, nodeInfoMap[node.Name]) if status.IsSuccess() != tt.fits[node.Name] { t.Errorf("[%s]: expected %v got %v", node.Name, tt.fits[node.Name], status.IsSuccess()) } } }) } } func TestPodTopologySpreadFilter_MultipleConstraints(t *testing.T) { tests := []struct { name         string pod          *v1.Pod nodes        []*v1.Node existingPods []*v1.Pod fits         map[string]bool }{ { // 1. to fulfil "zone" constraint, incoming pod can be placed on any zone (hence any node) // 2. to fulfil "node" constraint, incoming pod can be placed on node-x // intersection of (1) and (2) returns node-x name: "two constraints on zone and node, spreads = [3/3, 2/1/0/3]", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": false, "node-x": true, "node-y": false, }, }, { // 1. to fulfil "zone" constraint, incoming pod can be placed on zone1 (node-a or node-b) // 2. to fulfil "node" constraint, incoming pod can be placed on node-x // intersection of (1) and (2) returns no node name: "two constraints on zone and node, spreads = [3/4, 2/1/0/4]", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y4").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": false, "node-x": false, "node-y": false, }, }, { // 1. to fulfil "zone" constraint, incoming pod can be placed on zone2 (node-x or node-y) // 2. to fulfil "node" constraint, incoming pod can be placed on node-b or node-x // intersection of (1) and (2) returns node-x name: "constraints hold different labelSelectors, spreads = [1/0, 1/0/0/1]", pod: st.MakePod().Name("p").Label("foo", "").Label("bar", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("bar").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("bar", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": false, "node-x": true, "node-y": false, }, }, { // 1. to fulfil "zone" constraint, incoming pod can be placed on zone2 (node-x or node-y) // 2. to fulfil "node" constraint, incoming pod can be placed on node-a or node-b // intersection of (1) and (2) returns no node name: "constraints hold different labelSelectors, spreads = [1/0, 0/0/1/1]", pod: st.MakePod().Name("p").Label("foo", "").Label("bar", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("bar").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-x1").Node("node-x").Label("bar", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("bar", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": false, "node-x": false, "node-y": false, }, }, { // 1. to fulfil "zone" constraint, incoming pod can be placed on zone1 (node-a or node-b) // 2. to fulfil "node" constraint, incoming pod can be placed on node-b or node-x // intersection of (1) and (2) returns node-b name: "constraints hold different labelSelectors, spreads = [2/3, 1/0/0/1]", pod: st.MakePod().Name("p").Label("foo", "").Label("bar", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("bar").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Label("bar", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("foo", "").Obj(), st.MakePod().Name("p-y2").Node("node-y").Label("foo", "").Label("bar", "").Obj(), st.MakePod().Name("p-y3").Node("node-y").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": false, "node-b": true, "node-x": false, "node-y": false, }, }, { // 1. pod doesn't match itself on "zone" constraint, so it can be put onto any zone // 2. to fulfil "node" constraint, incoming pod can be placed on node-a or node-b // intersection of (1) and (2) returns node-a and node-b name: "constraints hold different labelSelectors but pod doesn't match itself on 'zone' constraint", pod: st.MakePod().Name("p").Label("bar", ""). SpreadConstraint(1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("bar").Obj()). Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("zone", "zone1").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("zone", "zone1").Label("node", "node-b").Obj(), st.MakeNode().Name("node-x").Label("zone", "zone2").Label("node", "node-x").Obj(), st.MakeNode().Name("node-y").Label("zone", "zone2").Label("node", "node-y").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-x1").Node("node-x").Label("bar", "").Obj(), st.MakePod().Name("p-y1").Node("node-y").Label("bar", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": true, "node-x": false, "node-y": false, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { nodeInfoMap := schedulernodeinfo.CreateNodeNameToInfoMap(tt.existingPods, tt.nodes) meta := predicates.GetPredicateMetadata(tt.pod, nodeInfoMap) state := v1alpha1.NewCycleState() state.Write(migration.PredicatesStateKey, &migration.PredicatesStateData{Reference: meta}) plugin, _ := New(nil, nil) for _, node := range tt.nodes { status := plugin.(*PodTopologySpread).Filter(context.Background(), state, tt.pod, nodeInfoMap[node.Name]) if status.IsSuccess() != tt.fits[node.Name] { t.Errorf("[%s]: expected %v got %v", node.Name, tt.fits[node.Name], status.IsSuccess()) } } }) } } </ins>

What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: __salt__['cmd.run']( 'ip link set dev {0} up'.format(name)) new_state = get_current_state() <del> if not new_state['iptables_rule_exists']: __salt__['iptables.append'](**iptables_rule) </del> <ins> if not new_state['iptables_rule_1_exists']: __salt__['iptables.append'](**iptables_rule_1) if not new_state['iptables_rule_2_exists']: __salt__['iptables.append'](**iptables_rule_2) </ins> new_state = get_current_state() ret['comment'] = 'The state of "{0}" was changed!'.format(name)

Spin up a cluster. Create a container on each of two machines. Try to ping each other's IPs. No love. It seems we need an extra iptables rule. For example, for a minion with the 10.244.2.0/24 subnet assigned, we need: iptables -t nat -A POSTROUTING -s 10.244.2.0/24 ! -d 10.244.2.0/24 -j MASQUERADE
Hrm, I can't repro this. I tried: ping 10.244.2.1 from a different kubernetes VM and it worked. (or do you mean from inside a container?)
From within a container to another container
repro'ed it for me, too.
Is this just ping? Or is this any network traffic? , Tim Hockin wrote:
Any traffic - the kernel is not masquerading it because our iptables magic doesn't match packets coming from inside a container on a veth. I saw this while trying to do another round of perf tests.


Question: - [ 'chroot', '{root}', '/bin/sh', '-c', 'DEBIAN_FRONTEND=noninteractive dpkg --install /tmp/docker.deb' ] - [ 'rm', '{root}/tmp/docker.deb' ] <ins> # Fix a cloud-init bug where it uses nobootwait # see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=789884 - [ 'chroot', '{root}', '/bin/sh', '-c', 'echo "mount_default_fields: [~, ~, ''auto'', ''defaults,nofail'', ''0'', ''2'']" > /etc/cloud/cloud.cfg.d/99_kubernetes.cfg' ] </ins> # We perform a full replacement of some grub conf variables: #   GRUB_CMDLINE_LINUX_DEFAULT (add memory cgroup) #   GRUB_TIMEOUT (remove boot delay)

Master failed to restart on Debian Jessie. Console logs showed: I have no idea where nobootwait is coming from though...
Aha... it's a bug in jessie's cloud-init. I have a fix, will produce a PR once verified.
Fix in


Question: // short-circuit on verflag verflag.PrintAndExitIfRequested() <del> cliflag.PrintFlags(cleanFlagSet) </del> // set feature gates from initial flags-based config if err := utilfeature.DefaultMutableFeatureGate.SetFromMap(kubeletConfig.FeatureGates); err != nil {

When starting the Kubelet with , the Kubelet begins with unstructured logging and then transitions to structured logging: I expect the Kubelet to have structured logging through the startup lifecycle. Kubernetes version (use ): master [] Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted /sig node
Is it possible that these components are not yet migrated? I don't think or are on structured logging yet.
/cc
It's true tha and and is not yet log structured, At least I don't see it in the PR that exists
This bug is related to JSON implementation, not structured logging. Moment that logging format configuration is applied happens late in component bootstrap process. (EDIT: changed the "Configuration of JSON" to "Moment that logging format configuration is applied")
If so, is it related to this PR ? This PR changes the from component-base/logs to the component-base/config
No, your PR is not related. As a configuration of JSON I meant the moment the configuration is applied and klog switches to use different format. This is done exactly when particular component calls . Fixing this issue should be done by changing component bootstrap process in a way that prioritizes configuring logging over any other configuration.
Thanks, I get it
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/milestone v1.22
/sig instrumentation /wg structured-logging
Hey there ( Bug-Triage here , Given the fact that the code freeze for this release already started, I´d like to suggest moving this issue to the next milestone. wdyt?
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
This needs to be fixed in kubelet. Those log messages that occur in text format should be moved to a point during init where logging is initialized.
Hey there ( Bug-Triage here , Just want to make sure this issue is still on track for the 1.23?

Hey what's the current status of init logs after your recent improvements? Is there anything left to do here?
No, this is not fixed. is still being called before . The behavior is a bit different, though: while previously it used the wrong default log format, now it uses the wrong default log level and therefore those messages are not printed. I'd also like to clarify one thing: the "Flag ... has been deprecated, will be removed in a future version" messages are printed by pflag to stderr. Those are not normal log messages and no matter what we do with logging, they will always be plain text.
The fix is easy. Whether there's anyone willing to approve a PR is a different question...
/milestone v1.24
We can still fix this in 1.23. PR is here:


Question: cloud := &fake_cloud.FakeCloud{} cloud.Region = region client := &testclient.Fake{} <del> controller := New(cloud, client, clusterName) </del> <ins> controller := New(cloud, client, "test-cluster") </ins> controller.init() cloud.Calls = nil    // ignore any cloud calls made in init() client.Actions = nil // ignore any client calls made in init()

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: podsDir string) (volumetypes.GeneratedOperations, error) { var pluginName string <del> if useCSIPlugin(og.volumePluginMgr, volumeToUnmount.VolumeSpec) { </del> <ins> if volumeToUnmount.VolumeSpec != nil && useCSIPlugin(og.volumePluginMgr, volumeToUnmount.VolumeSpec) { </ins> pluginName = csi.CSIPluginName } else { pluginName = volumeToUnmount.PluginName

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: LocalVolumeType: gce-localssd-scsi-fs Since when has it been failing: 2/20 Testgrid link: Reason for failure: unsure yet /assign
After , there are still two failing tests: For the first one, looks like reconstruction is not correctly implemented for local PVs: to check if it's ok for mounter to be initialized with an empty path the panic in the operation_generator
cc Unsure why it's failing only for this local pv type (real disk vs loopback block) I have not investigated the 2nd test yet
/priority critical-urgent /milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/assign
hi, checked , after , "should be mountable" has been fixed. failing test is "should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]" and it fails for all kinds of local volume types
This is because local volume cannot be reconstructed for now (localPlugin.NewMounter requires field is set), so is going to . But uses to , and , so it panics here. I'm not sure how to support reconstruction for local volumes correctly yet. But we can fix this panic first, then reconciler will clean up the existing mount points if the volume is no longer needed. IIUC, this will fix this failing test. After this is fixed, if necessary we can revisit how to reconstruct local volumes correctly.
I see similar failures in and too; are they connected? cc
yes they should be similar
need to keep track of this to see if failures in and also clear up.
The tests are still throwing s in (it's the job there though). I've opened to track the failures in 1.14-blocking. They also seem to not have run since the 24th, I'll look into that in the meantime


Question: Returns a JSON with the fields `output` (containing the file's name on the server) and `error` containing any potential server side errors. <ins> If `--tls-cert-file` is added (ideally in conjunction with `--tls-private-key-file`, the HTTP server will be upgraded to HTTPS. The image has default, `localhost`-based cert/privkey files at `/localhost.crt` and `/localhost.key` (see: [`porter` subcommand](#porter)) </ins> It will also start a UDP server on the indicated UDP port that responds to the following commands: - `hostname`: Returns the server's hostname

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: } } <ins> func TestResizeController(t *testing.T) { fakeClient := FakeKubeClient{} name := "name" replicas := 17 ResizeController(name, replicas, &fakeClient) if len(fakeClient.actions) != 2 { t.Errorf("Unexpected actions: %#v", fakeClient.actions) } if fakeClient.actions[0].action != "get-controller" || fakeClient.actions[0].value.(string) != name { t.Errorf("Unexpected action: %#v", fakeClient.actions[0]) } controller := fakeClient.actions[1].value.(ReplicationController) if fakeClient.actions[1].action != "update-controller" || controller.DesiredState.Replicas != 17 { t.Errorf("Unexpected action: %#v", fakeClient.actions[1]) } } </ins> func TestCloudCfgDeleteController(t *testing.T) { fakeClient := FakeKubeClient{} name := "name"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD (latest) Environment: dind What happened: Enpoints object is constantly getting updated every 2 seconds, even though nothing has changed in the object. It will produce lot of traffic on any informer watching on endpoints. An update event is produced every 2 second per endpoint. What you expected to happen: Endpoints(any object) should be updated only when there is change in either spec or status. How to reproduce it (as minimally and precisely as possible): Just watch on endpoints Also see the logs from federation controller manager, which is watching on endpoints updates from federated cluster Anything else we need to know: Looking at the endpoints controller code . it looks like reflect.DeepEqual is not working for endpoints labels and Subsets (may be need sorting, just a guess). I also see another issue raised by some time back , related but slightly different. I think this is a bug and if fixed will reduce lot of traffic to etcd.
Those two endpoints specifically are used as shared lock objects by controllers. Agree that's a bad object to use because so many things watch them for updating iptables, etc
code in question:
dupe of


Question: } func updateISCSINode(b iscsiDiskMounter, tp string) error { <ins> // setting node.session.scan to manual to handle https://github.com/kubernetes/kubernetes/issues/90982 out, err := execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.scan", "-v", "manual") if err != nil { // don't fail if iscsiadm fails or the version does not support node.session.scan - log a warning to highlight the potential exposure klog.Warningf("iscsi: failed to update node with node.session.scan=manual, possible exposure to issue 90982: %v", out) } </ins> if !b.chapSession { return nil } <del> out, err := execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.auth.authmethod", "-v", "CHAP") </del> <ins> out, err = execWithLog(b, "iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", "node.session.auth.authmethod", "-v", "CHAP") </ins> if err != nil { return fmt.Errorf("iscsi: failed to update node with CHAP, output: %v", out) }

What happened: The issue stems from use cases were 1 initiator is used for multiple LUNs. Because of this, all of the LUNs are discovered and mapped on a node when it does its initial discovery / login via AttachDisk[1]. Because a pod never mounts the device on any of the other nodes, it is never removed. [1] In situations where LUN numbers can be reused, it’s possible to hit the below race condition where a new LUN reattaches to a faulty multipath device created for a previous LUN of the same number. What you expected to happen: Only the required LUN should be discovered and mounted. How to reproduce it (as minimally and precisely as possible): Assume we start from an empty environment using Node A and B. An empty environment is not a requirement, it just makes it visually easier. 1) User A creates a PVC, the PV and LUN1 are dynamically provisioned and mounted by the pod on Node A. Node A does an iscsiadm login (in addition to other things) and scans for the scsi devices, adds them to the multipath devicemap and attaches to /dev/dm-XX. 2) User B creates a second PVC, the PV and LUN2 are dynamically provisioned and mounted by the pod on Node B. Node B does an iscsiadm login (in addition to other things) and scans for the scsi devices, adds them to the multipath devicemap and attaches to a /dev/dm-XX device. This process finds both LUN1 and LUN2 and creates multipath devicemaps and devices for both of them. 3) User A creates a PVC, the PV and LUN3 are dynamically provisioned and mounted by the pod on Node A. Node A is already logged in because of LUN1 so it just scans for the new LUN, adds it to a multipath devicemap and attaches to a /dev/dm-XX device.

At this point: Node A knows about LUN1 and has it mounted and LUN3 and has it mounted Node B knows about LUN1 and LUN2 but only has LUN2 mounted 4) User A deletes their first pod and PVC from Node A. This triggers an unmount and the flushing and removal of the multipath device and underlying block devices on Node A. This deletes LUN1 on the backend. Because the pod was only running on Node A, the unmount and removal procedures do nothing for the discovered LUN1 and its devices on Node B. The multipath device for LUN1 on Node B loses its paths and goes into a faulty state. 5) User C creates a PVC, the PV and LUN1 (reused number) are dynamically provisioned and mounted by the pod on Node B. Node B is already logged in because of LUN2 and it already knows about LUN1 as it was discovered when LUN2 came along. In this case, LUN1 is reattached to the old multipath device and block devices from the previously deleted LUN1. This is when we see the LUN assignment / remap warning: Ex: May 5 10:02:58 server kernel: sd 40:0:0:29: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical 6) Kubernetes tries to mount the multipath device to its pod and, based on our testing, sees a variety of possible issues, depending on what kind of leftover data is visible. These range from fixable fsck file system issues, fsck failures and complete disk formatting failures. The other primary issue we see is where there are PVC size mismatches and non fatal fsck fixes. If LUN1 was originally 1GB, the multipath and block devices reflect that. If the new PVC asked for 5GB, LUN1 is 5GB, but when it reconnects, the user only sees the previous 1GB devices. The reverse is also true where you could request 1GB and end up with the old 5GB block devices which results in formatting errors. Anything else we need to know?: The race condition is somewhat uncommon and depends on LUN number reuse and WWID stings.

As the number of LUNs grow and as more servers are restarted for patching etc.. there will be a great and greater chance of people hitting this as churn occurs. The other big issue which is independent of hitting the above race condition is the performance impact and stability of the server. Having a few hundred extra devices is not ideal, but the bigger issue looks to be the constant failed i/o’s and multipathd path down checks for LUNs which have been removed and remapped. Environment: kubernetes Kubernetes version (use ): master Cloud provider or hardware configuration: all OS (e.g: ): all Kernel (e.g. ): all
/sig storage
/assign


Question: // A pod is besteffort if none of its containers have specified any requests or limits. // A pod is guaranteed only when requests and limits are specified for all the containers and they are equal. // A pod is burstable if limits and requests do not match across all containers. <ins> // When this function is updated please also update staging/src/k8s.io/kubectl/pkg/util/qos/qos.go </ins> func GetPodQOS(pod *core.Pod) core.PodQOSClass { requests := core.ResourceList{} limits := core.ResourceList{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubectl library, logic is not the same as the one in What you expected to happen: logic should be the same. How to reproduce it (as minimally and precisely as possible): Create a pod with a guaranteed QoS class and a best-effort QoS initcontainer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
It seems to be duplicate functions here.
The intention for duplicates is coming from the fact that kubectl can't rely on packages from kuberentes main repository. Either way they should both be identical except for the fact that the internval version will use internal APIs, whereas kubectl's version will use versioned.

feel free to open a PR updating kubectl version updating it to match both versions.
/triage accepted /assign
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return pv, nil } <ins> func (p *glusterfsVolumeProvisioner) GetClusterNodes(cli *gcli.Client, cluster string) (dynamicHostIps []string, err error) { clusterinfo, err := cli.ClusterInfo(cluster) if err != nil { glog.Errorf("glusterfs: failed to get cluster details: %v", err) return nil, fmt.Errorf("failed to get cluster details: %v", err) } // For the dynamically provisioned volume, we gather the list of node IPs // of the cluster on which provisioned volume belongs to, as there can be multiple // clusters. for _, node := range clusterinfo.Nodes { nodei, err := cli.NodeInfo(string(node)) if err != nil { glog.Errorf("glusterfs: failed to get hostip: %v", err) return nil, fmt.Errorf("failed to get hostip: %v", err) } ipaddr := dstrings.Join(nodei.NodeAddRequest.Hostnames.Storage, "") dynamicHostIps = append(dynamicHostIps, ipaddr) } glog.V(3).Infof("glusterfs: hostlist :%v", dynamicHostIps) if len(dynamicHostIps) == 0 { glog.Errorf("glusterfs: no hosts found: %v", err) return nil, fmt.Errorf("no hosts found: %v", err) } return dynamicHostIps, nil } </ins> func (p *glusterfsVolumeProvisioner) CreateVolume(gid int) (r *v1.GlusterfsVolumeSource, size int, err error) { var clusterIDs []string capacity := p.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: plugin:          plugin, MetricsProvider: volume.NewMetricsStatFS(getPath(pod.UID, spec.Name(), plugin.host)), }, <del> server:       source.Server, </del> <ins> server:       getServerFromSource(source), </ins> exportPath:   source.Path, readOnly:     readOnly, mountOptions: util.MountOptionFromSpec(spec),

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign