Question: valid1 := map[string]string{ "Corefile": fmt.Sprintf(`.:53 { health <ins> ready </ins> kubernetes %v in-addr.arpa ip6.arpa { pods insecure upstream

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-network] DNS configMap nameserver [IPv4] Change stubDomain should be able to change stubDomain configuration [Slow][Serial] [sig-network] DNS configMap nameserver [IPv4] Forward external name lookup should forward externalname lookup to upstream nameserver [sig-network] DNS configMap nameserver [IPv4] Forward PTR lookup should forward PTR records lookup to upstream nameserver [Slow][Serial] Since when has it been failing: These tests have been failing since 5/18. Testgrid link: Reason for failure: All of these have this failure message in common: The failures coincide with for which the possible fix is . However, we are filing these failures separate from to see if there are any additional steps that need to be done for these series of tests. /cc /priority critical-urgent /kind failing test /milestone v1.15 /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved
Tests have been green since 6/1


Question: # Connecting to applications: kubectl proxy and apiserver proxy <ins> <!-- BEGIN MUNGE: GENERATED_TOC --> - [Connecting to applications: kubectl proxy and apiserver proxy](#connecting-to-applications-kubectl-proxy-and-apiserver-proxy) - [Getting the apiserver proxy URL of kube-ui](#getting-the-apiserver-proxy-url-of-kube-ui) - [Connecting to the kube-ui service from your local workstation](#connecting-to-the-kube-ui-service-from-your-local-workstation) <!-- END MUNGE: GENERATED_TOC --> </ins> You have seen the [basics](accessing-the-cluster.md) about `kubectl proxy` and `apiserver proxy`. This guide shows how to use them together to access a service([kube-ui](ui.md)) running on the Kubernetes cluster from your workstation.

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Question: # tidy staging repos in reverse dependency order. # the content of dependencies' go.mod files affects what `go mod tidy` chooses to record in a go.mod file. tidy_unordered="${TMP_DIR}/tidy_unordered.txt" <del> kube::util::list_staging_repos | xargs -I {} echo "k8s.io/{}" > "${tidy_unordered}" </del> <ins> kube::util::list_staging_repos  | xargs -I {} echo "k8s.io/{}" > "${tidy_unordered}" </ins> rm -f "${TMP_DIR}/tidy_deps.txt" # SC2094 checks that you do not read and write to the same file in a pipeline. # We do read from ${tidy_unordered} in the pipeline and mention it within the

I ran update- Not to see a ton of errors Kubernetes version (use ): HEAD of master
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture (code organization / dependencies) /sig testing (hack scripts)
/triage accepted


Question: utilnet "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> utilwait "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apiserver/pkg/admission" genericapiserver "k8s.io/apiserver/pkg/server" "k8s.io/apiserver/pkg/server/filters" serverstorage "k8s.io/apiserver/pkg/server/storage" "k8s.io/kubernetes/cmd/kube-apiserver/app/options" <ins> "k8s.io/kubernetes/cmd/kube-apiserver/app/preflight" </ins> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/capabilities"

See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).


Question: #	[ARCH=amd64] [CNI_RELEASE=v0.3.0] make (build|push) ARCH?=amd64 <del> CNI_RELEASE?=9d5e6e60e79491207834ae8439e80c943db65a69 </del> <ins> CNI_RELEASE?=07a8a28637e97b22eb8dfe710eeae1344f69d16e </ins> CNI_TARBALL=cni-$(ARCH)-$(CNI_RELEASE).tar.gz CUR_DIR=$(shell pwd) OUTPUT_DIR=$(CUR_DIR)/output

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: type RBDUtil struct{} func (util *RBDUtil) MakeGlobalPDName(rbd rbd) string { <del> return makePDNameInternal(rbd.plugin.host, rbd.pool, rbd.image) </del> <ins> return makePDNameInternal(rbd.plugin.host, rbd.Pool, rbd.Image) } func (util *RBDUtil) rbdLock(rbd rbd, lock bool) error { var err error var output, locker string var cmd []byte var secret_opt []string if rbd.Secret != "" { secret_opt = []string{"--key=" + rbd.Secret} } else { secret_opt = []string{"-k", rbd.Keyring} } // construct lock id using host name and a magic prefix lock_id := "kubelet_lock_magic_" + node.GetHostname("") l := len(rbd.Mon) // avoid mount storm, pick a host randomly start := rand.Int() % l // iterate all hosts until mount succeeds. for i := start; i < start+l; i++ { mon := rbd.Mon[i%l] // cmd "rbd lock list" serves two purposes: // for fencing, check if lock already held for this host // this edge case happens if host crashes in the middle of acquiring lock and mounting rbd // for defencing, get the locker name, something like "client.1234" cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "list", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) output = string(cmd) if err != nil { continue } if lock { // check if lock is already held for this host by matching lock_id and rbd lock id if strings.Contains(output, lock_id) { // this host already holds the lock, exit glog.V(1).Infof("rbd: lock already held for %s", lock_id) return nil } // hold a lock: rbd lock add cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "add", rbd.Image, lock_id, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) } else { // defencing, find locker name ind := strings.LastIndex(output, lock_id) - 1 for i := ind; i >= 0; i-- { if output[i] == 'n' { locker = output[(i + 1):ind] break } } // remove a lock: rbd lock remove cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "remove", rbd.Image, lock_id, locker, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) } if err == nil { //lock is acquired break } } return err } func (util *RBDUtil) persistRBD(rbd rbd, mnt string) error { file := path.Join(mnt, "rbd.json") fp, err := os.Create(file) if err != nil { return fmt.Errorf("rbd: create err %s/%s", file, err) } defer fp.Close() encoder := json.NewEncoder(fp) if err = encoder.Encode(rbd); err != nil { return fmt.Errorf("rbd: encode err: %v.", err) } return nil } func (util *RBDUtil) loadRBD(rbd *rbd, mnt string) error { file := path.Join(mnt, "rbd.json") fp, err := os.Open(file) if err != nil { return fmt.Errorf("rbd: open err %s/%s", file, err) } defer fp.Close() decoder := json.NewDecoder(fp) if err = decoder.Decode(rbd); err != nil { return fmt.Errorf("rbd: decode err: %v.", err) } return nil } func (util *RBDUtil) fencing(rbd rbd) error { // no need to fence readOnly if rbd.ReadOnly { return nil } return util.rbdLock(rbd, true) } func (util *RBDUtil) defencing(rbd rbd) error { // no need to fence readOnly if rbd.ReadOnly { return nil } return util.rbdLock(rbd, false) </ins> } func (util *RBDUtil) AttachDisk(rbd rbd) error { var err error <del> devicePath := strings.Join([]string{"/dev/rbd", rbd.pool, rbd.image}, "/") </del> <ins> devicePath := strings.Join([]string{"/dev/rbd", rbd.Pool, rbd.Image}, "/") </ins> exist := waitForPathToExist(devicePath, 1) if !exist { // modprobe

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: errorf(fmt.Sprintf("Failed to find jump to lb chain %v", lbChain), kubeNodePortRules, t) } <ins> if !hasJump(kubeNodePortRules, string(KubeMarkMasqChain), "", svcNodePort) { errorf(fmt.Sprintf("Failed to find jump to %s chain for destination IP %d", KubeMarkMasqChain, svcNodePort), kubeNodePortRules, t) } </ins> svcChain := string(servicePortChainName(svcPortName.String(), proto)) lbRules := ipt.GetRules(lbChain) if hasJump(lbRules, nonLocalEpChain, "", 0) {

/kind bug /sig network If I set up a service with a NodePort and set , and try to access it as it works. If I try to access it as , it triggers a martian source error - the packet gets forwarded as 127.0.0.1. We need some masquerade in there. Applies to iptables mode. Maybe ipvs mode, not sure?
If we set for NodePort, we don't do any masq for packets currently. I have tested that masq will resolve the issue in iptables mode, but my question is "do we still need to preserve source ip of packets for nodeport when externalTrafficPolicy=Local"? If we do masq, we will lose the source IP, sadly.
BTW, with the fix of , is reachable The iptables created by is Any comments?
this is not entirely true. key "externalTrafficPolicy=Local" needs to be a solution to the problem of obtaining the real IP of the client, but this is not happening. If we do masq, we will not lose the source IP, because now he doesn't work (version 1.8.1) the problem with the inaccessibility of the actual source IP () in version 1.8.1 is still relevant and is a very big problem for deployment in kubernetes most web services on a production. besides, in my cluster key "externalTrafficPolicy : false" leads to a hollow stop traffic to the pods.
Without masq, nodeport for localhost when set "externalTrafficPolicy=Local" won't work. If service is not reachable, say "how to preserve source IP" won't make sense. Listen, masq is used for someone who want to visit nodeport via localhost when set "externalTrafficPolicy=Local" and don't require preserving source IP. If source IP is really important to you, you can just ignore the .
I do not understand what do you mean "you can just ignore the 127.0.0.1". could you describe in detail how to configure the service to get the real source IP?

type service, externalTrafficPolicy=[Local | Cluster] type service, externalTrafficPolicy=Local, but don't try to access type service, externalTrafficPolicy=Local
i wrote above, NodePort type service with externalTrafficPolicy=Local blocking the transmission of traffic to the pod. I'm trying to connect to node_ip:port. maybe this is a bug? I could demonstrate this.
oh, sorry )) i'm connecting to wrong node )) yes, now traffic is gone. but ip is not source! with "externalTrafficPolicy=Cluster" i getting ip 10.1.29.21 (node ip) with "externalTrafficPolicy=Local" i getting ip 10.4.62.1 (pods network ip - not real source ip)
You probably should check the iptables applied in your host, is there any masq rules created by other processes other than kube-proxy? It preserves the source IP of client - I double checked it. BTW, you can create another thread for discussing your problems. I feel like this issue is orthogonal with your question.
Will setting ClusterIP type service and externalTrafficPolicy=Local for a service to a DaemonSet, route traffic only to the daemonSet pod in that node when i access the service using DaemonSetServiceIP?
no. local service does not work for cluster IP


Question: } return retErr } <ins> func base64decode(v string) (string, error) { data, err := base64.StdEncoding.DecodeString(v) if err != nil { return "", fmt.Errorf("base64 decode failed: %v", err) } return string(data), nil } </ins>

Currently if one wants to view secrets stored in k8s they must follow the process described here: This involves copy&pasting. It seems to me that kubectl could provide a more helpful interface to base64 decrypt the users on behalf of the user saving them the trouble.
The kubelet does decode secrets before exposing them to the pod. Also the title doesn't really make sense. Secrets aren't encrypted so there's nothing to decrypt.
/kind feature
Yes, though this is low priority since we mostly don't want end users to view these once they get into the system.
Why can't that be an API permissions issue? I mean, if I really want to see it, I can just launch a pod to tell me.
I've update the issue title with what I understand as the request. To the best of my understanding this is just a request for us to avoid the need to pipe into base64 when inspecting secrets.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
I would like to try to make this work before it rots. cc Any guidance on a preferred implementation?
What do you think about adding support for functions in templates? This would allow someone to use the existing and a template that uses the function -- or a wide variety of other interesting functions as a side effect. I believe (but have not yet tested) that the following might be sufficient to enable the requested behaviour (plus the change to the dependencies list, of course):
Hi I suppose my thoughts on this issue have changed a little since I wrote it. The situation isn't quite as bad as I described as it is possible to decode a secret in one go without then need to copy and paste. i.e.

That said, that is a bit of a mouthful. Also, it might be nice to more easily 'work' with individual secrets and set them as well as get them. Should a user really need to pull down an entire group of secrets and edit a file just to be able to update a single password? For example something like this might be possible: What do people think? PS: Should this have been an issue on [kubectl] instead? PPS: I think some of the ugliness of this comes from configmaps being very similar to secrets except for the base64 encoding. Have there been any thoughts around bringing them into line? If memory serves, it might actually be nicer, due to YAML syntax-shenanigans, if configmaps were base64 encoded too.) [kubectl]:
Hi ... yes, I was thinking that it would be handy to be able to do something like: which is essentially the same as what you did without the overhead of the base64 process, or embed the information in a larger template like or in theory be able to use any of the other Sprig functions if someone found them useful. I think that having the Sprig methods available for generic formatting would just be useful on its own, really. On the topic of a or a , that seems pretty cool. There seem to be quite a lot of options for that I think people might expect to see mirrored in a if it were to exist. Thoughts?
It's also come to my attention that works great on MacOS but on Linux it's , so if someone, say, has a Helm chart where the notes tell the user to retrieve the secret data using one or the other of these, it won't work on the other one. would clearly be a simpler experience for the user than Hmm... thinking further: does does the idea conflict with the documentation in ? Perhaps we'd have to do something like: or similar?
/remove-lifecycle rotten
Small tip: you can use which works on both Linux and Mac.
Thank you
Update: I've looked at the huge amount of additional code that would be imported as a result of importing sprig and decided to go with a much smaller change instead, which is simply to implement a template function using . See PR with code and test.
please see the linked PR which addresses the original issue in a minimal fashion -- it simply provides a template function.

It has passed all the tests and needs review; if it's good enough to solve the original problem I'm happy to squash the commits and can hope to get it merged. As far as solving evolved description of the problem, I'm not sure where to start or if you really want me to go in that direction. Would you consider taking this change and splitting the additional work for being able to get / set the contents of an individual key in a secret into a separate issue? If you have any guidance on what you'd like to see with that I'd be happy to take a look, but it might be out of my current grasp.


Question: baseName:       tt.fields.baseName, certificateDir: tt.fields.certificateDir, } <del> if got := rw.Exists(); got != tt.want { </del> <ins> if got, _ := rw.Exists(); got != tt.want { </ins> t.Errorf("pkiCertificateReadWriter.Exists() = %v, want %v", got, tt.want) } })

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: <ins> rules: - selectorRegexp: k8s[.]io/kubernetes allowedPrefixes: - k8s.io/kubernetes/cmd/kube-apiserver - k8s.io/kubernetes/pkg - k8s.io/kubernetes/plugin - k8s.io/kubernetes/test/utils - k8s.io/kubernetes/third_party </ins>

found this by running tests on this PR: it's problematic for a couple of reasons: prevents kubeadm from moving out (eventually) there are no API guarantees in the given code/structs/functions in pkiutils the logic in cmd/kube- must be reworked to generate cert/key pair without using kubeadm. originally in pending fix PR
/sig api-machinery cluster-lifecycle architecture /area code-organization
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery sig-cluster-lifecycle architecture /area code-organization </details
/assign
cc
Please make sure to add an import restriction to prevent a regression in the future.
/triage accepted


Question: s := &gRPCService{callTimeout: callTimeout} s.connection, err = grpc.Dial( addr, <ins> grpc.WithAuthority("localhost"), </ins> grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithDefaultCallOptions(grpc.WaitForReady(true)), grpc.WithContextDialer(

KMS Plugins require communication via grpc UDS. Currently, the k8s grpc client sets the authority header to the socket path which is marked as invalid preventing successful socket communication. I expect a custom KMS provider plugin to be sent valid authority headers so that successful communication can be established between the k8s and grpc client/server. Create a grpc server according to the and attempt to establish socket communication. This issue has been resolved in for device plugins and CSI. The same solution will likely fix this issue as well. I will create a PR for this and link it here after. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig auth /triage accepted


Question: "github.com/davecgh/go-spew/spew" "github.com/ghodss/yaml" <ins> "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/util/mergepatch" )

Extracted from The RawExtension type allows an API object to inline arbitrary JSON. strategicpatch.CreateTwoWayMergePatch returns an error trying to map that inlined JSON to struct fields. It should be able to compute a patch for an object containing a field of that type.
cc
We need to decide what strategic merge patch should do in the following scenarios: unknown fields (can be encountered when the API or a user-defined object has a new field the compiled-in struct doesn't have yet) unmergeable fields (like RawExtension) cc
The current behavior is: unknown fields: trying to merge a unknown field will fail when lookup the metadata in the struct tags unmergeable fields: will be replaced as raw byte array Maybe we can use generic json merge for unmergeable fields?
Did that change since 1.5? In 1.5, we were getting an error during patch computation for objects with a RawExtension field
Sorry. This behavior is in 1.4.7. e.g . It will error in 1.5.
Yeah, the field gets replaced. Even in 1.4.7, an unknown field (like initcontainers in pods) or an object including a RawExtension (anything with a field of type or ) would error
My personal favor would be to instruct the patch to ignore such field, seems the cleanest approach to me. Similarly to how we allow annotating deletion and merging.
wouldn't it be confusing to submit a patch with stuff in that field, have the patch accepted, and have none of your submitted changes take effect? seems like the least surprising thing would be to treat struct fields like or as
How about both, iow. having as a default and as an opt-in. This would allow the resource author to decide how his resource should be treated. Eventually on we could return the ignored fields. Having said that we could start with right away. I think I could work on a patch implementing that if majority agrees (
I think is a good place to start when computing/applying patches for known struct fields like RawExtension. I'll open a separate issue for client-side handling of computing patches for unknown fields
Ignore the field sounds like a really good solution. WDYT of printing a warning to stdeer?
what does ignoring instead of replacing get us?
Sorry I was reading the comment on my phone between other things. Ignore my comment.


Question: Filter: v1beta2.PluginSet{ Enabled: []v1beta2.Plugin{ {Name: "Plugin2", Weight: pointer.Int32Ptr(2)}, <ins> {Name: "Plugin1", Weight: pointer.Int32Ptr(1)}, </ins> }, }, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Recent commit: import an issue when multiple custom plugin configured in scheduler config file, and those plugins are default plugins too. It was manipulating slice with index, but the slice is updated while index is not, which will cause the index out of range, see the exceptions listed below, config file should be handled properly without such exception override multiple default plugin in scheduler's config file. Kubernetes version (use ): latest (1.22) Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign
/sig scheduling /triage accepted


Question: inList $etcdName $myIP configEtcd $etcdName $myIP $cluster # For minion set MINION IP in default_scripts/kubelet <del> sed -i "s/MY_IP/${myIP}/g" default_scripts/kubelet sed -i "s/MASTER_IP/${masterIP}/g" default_scripts/kubelet sed -i "s/MASTER_IP/${masterIP}/g" default_scripts/kube-proxy </del> <ins> sed -i "s/MY_IP/${myIP}/g" work/default_scripts/kubelet sed -i "s/MASTER_IP/${masterIP}/g" work/default_scripts/kubelet sed -i "s/MASTER_IP/${masterIP}/g" work/default_scripts/kube-proxy </ins> # For master set MINION IPs in kube-controller-manager <del> if [ -z "$minionIPs" ]; then </del> <ins> if [ -z "$minionIPs" ]; then </ins> #one node act as both minion and master role minionIPs="$myIP" else minionIPs="$minionIPs,$myIP" fi <del> sed -i "s/MINION_IPS/${minionIPs}/g" default_scripts/kube-controller-manager </del> <ins> sed -i "s/MINION_IPS/${minionIPs}/g" work/default_scripts/kube-controller-manager </ins> cpMaster cpMinion

I used the ubuntu-cluster setup scripts to build a mini testing environment. The first time I inputted a wrong config and when I reconfigured the environment by running the scripts again an error occured. I looked into the code and found the scripts hadn't deal with old configs. And of kube-proxy configs are not replaced.
Looks like this was fixed by .


Question: } func TestPersistentVolumeRecycler(t *testing.T) { <ins> glog.V(2).Infof("TestPersistentVolumeRecycler started") </ins> _, s := framework.RunAMaster(t) defer s.Close()

Test failed with and the relevant part of the stack trace is: Logs:
(or anyone), integration tests do not log much and I don't know what's going on in volume controller. Would it be possible to raise integration test log level at least to v=3 (4 preferred)?
what is the status of
can you please take a look?
Increasing integration log level can go into I think. It's probably a good idea to split them into their own job.


Question: glog.V(5).Infof("findmnt: directory %s does not exist", file) return true, err } <del> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target", "--noheadings", "--target", file} </del> <ins> // Add --first-only option: since we are testing for the absense of a mountpoint, it is sufficient to get only // the first of multiple possible mountpoints using --first-only. // Also add fstype output to make sure that the output of target file will give the full path // TODO: Need more refactoring for this function. Track the solution with issue #26996 args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target,fstype", "--noheadings", "--first-only", "--target", file} </ins> glog.V(5).Infof("findmnt command: %v %v", nsenterPath, args) exec := exec.New()

Kubelet (v1.2.4) can consume a lot cycles and when it keeps trying to tear orphaned volumes down:
See also cc
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
/cc

I'm running Kubernetes v1.2.3 using the single-node instructions here: with Docker v1.10. Everything worked as expected for several hours, but now pods are not being scheduled or removed. I've attempted to update a deployment several times over the last hour. Here is for that deployment: The ContainerCreating pod's last event from is: When I delete a pod, the scheduler does seem to start the new pod almost immediately, and the kubelet logs: The kubelet then begins logging a readiness check failure every 10s: but I can't see any evidence that it actually tries to run the container. Docker doesn't emit any logs. The Docker daemon is responsive, and the image is definitely present. When I run the image using it works fine. Anything I can check? Thanks!
At a glance, looks like some bug in state transitions at the kubelet.
is there any message in the kubelet log related to image pulling? Without further information, it seems like kubelet could be stuck at pulling the images. It'd be helpful to see if there are related information.
Can you run docker ps -a to check if docker failed to create / run containers? Could you please upload related Kubelet log somewhere so that we can access. Based on the initial description, it looks like the image is successfully pulled.
I was just making a wild guess because image pulling is usually where the pod workers get stuck and become unresponsive for a long time :) From the description, the "cluster" was working for a few hours, but the pods were newly created.
and , does not show any attempted containers of the relevant image. Here are the logs of the kubelet. I d the existing pod in the deployment, which was in the state; and a new pod was immediately scheduled. These logs cover those events. The Readiness probes continue at 10s intervals. I sent you the full logs of the kubelet in a Slack pm.
just an update, the kubelet is still in the same state: Some context that may be helpful is this happened after I had been experimenting with deploying an NFS server using the current Kubernetes example, , and . I changed the replication controller at to a deployment, and updated it several times.

I was changing the code inside the container, so some of those attempts may have contained incorrect NFS configuration and/or failed to clean up completely before the container exited. I haven't seen this particular kubelet state before. I have seen slow pod state changes, , but have never seen them halt completely. This state is also different because the kubelet is not logging anything. Previously when I have seen slow pod state changes, it has logged repeatedly about orphaned volumes. Is it plausible that this issue has something to do with the state of the kernel after my experimentation with NFS?
/cc
If this is related to stuck image pulling, what you'll see in the docker daemon's log is something like this: If pulling is stuck, this is all that you'll see. If pulling isn't stuck, then you should see something like the following: And in the kubelet's log, this is the last entry you should expect to see for a pod that is stuck Pending / ContainerCreating if pulling is indeed stuck:
the image is definitely present on the node. I've verified by hand: and also the kubelet doesn't log about creating an image. The logs here: are all it says after a pod is scheduled. Note also that several pods have been in the 'Terminating' state for three days.
can you ssh to the node and curl the kubelet port to get a stack trace? e.g.
Stack trace attached.
goroutine 192 looks
What does it mean?
I think it's trying to mount the NFS directory. You said you had an incorrect configuration at one point?
There are several goroutines in this state
It's safe to assume I had an incorrect configuration at one point, yeah.
Wrong IP address for the NFS server, or ? , Anand Patil wrote:
To clarify, that deployment doesn't consume any NFS persistent volumes, and none of my versions did. It consumes a host volume and runs an NFS server. The pod spec is attached.
So it can't be hanging trying to mount an NFS volume, it's trying to create an NFS volume.
By "incorrect configuration", I meant that my version of this: could have been wrong.
Oh ok. Here's one of several hung pod workers: I don't know that this pod worker is necessarily associated with the nfs server pod. Regardless, it's trying to run And hanging.

You probably need to get some help from when they have time.
OK, I appreciate your attention to this issue, I look forward to hearing from them.
do you have installed on the Docker host?
xref
Hi unfortunately the machine where this issue was observed has been shut down. I saw something similar since then, though, and have a hunch about what was going on. A running pod had an NFS volume source, then the NFS server was shut down. The result was that there were some folders in in which recursive walkers like and would hang as described here: . I can try to create a simple repro case, if that would be helpful.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.

Kubelet continuously throws lots of errors: I'm starting the cluster this way:
To reproduce this behavior I cleaned up everything related to kubernetes (containers, /var/lib/kubernetes contents, etc), started hyperkube, then started a bunch of containers - everything is good up to this point. Then I delete a pod with (kubectl delete pod frontend), and this happens:
More observations. Here is my default token: Here is error message I get: This path is changing all the time: (Note the subdirectory names are different) Pod deleteion code was recently touched by authentication tokens (whose directories can't be removed) by Any ideas what it could be? I would ignore this problem if it didn't consume 15% of CPU in a very tight orphan removal loop.
Those tokens are just secrets as far as the kubelet is concerned. Not sure how secret volume cleanup differs in hyperkube. cc
We have same problem on nodes with CoreOS alpha 991.0.0 and hyperkube 1.2 a lot of messages like:
same issue here.
Same on Ubuntu 14.04, hyperkube v1.2.3, docker 1.9.1 with aufs backend, kernel 3.13.0-48-generic. I also see pod state transitions taking a long time; they hang in "Pending" and "Terminating" for minutes. Does anyone else see that in conjunction with this issue?
For context, this issue doesn't show up immediately in my hyperkube setup. It turns up after a few days. During that time, I update 6 or 7 deployments simultaneously, several times per day.
Sorry I missed the ping on this. Assigned to self, will try to triage by the end of the week.
cc
got the same problem on centos
I am wondering if this depends on Docker version, and not so much the graph driver.
xref
Same problem here, any ideas?
could you please re-triage this bug. I'm not sure "team/ux" is the right tag for it - it doesn't have anything common with UX. It looks like a problem with some low-level container machinery. And given the number of +1's, maybe raise the priority? Thanks.
What problem does this cause for you?
!

The error msg will be a confuse to log/alert monitor infrastructures and really hope it can be fixed.
As I wrote before: I have a 8-core laptop that can work ~ 4 hours on battery (enough for a mid-range flight). When I run Kubernetes cluster via hyperkube, it loads 15% of my CPU resources (more than 1 core!) in a tight loop draining my battery less than in hour.
Assigning (all seem related--secret volume tear down issues with hyperkube) to for investigation.
After setting up a hypekube and digging the logs, I think the following happened and caused the issue. Hyperkube automatically creates a few pods for dns, dashboard, proxy services etc. But there is something wrong when setting up and mounting the volumes. The following shows some logs from one such pod. (1) 21:20:03. ] SyncLoop (ADD, "api"): "kubernetes-dashboard-fnivxkube-system(-29d0-11e6-bafc-)" (2) 21:20:03. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-]  These two steps show that volume is mounted (3) 21:20:07. ] IsLikelyNotMountPoint findmnt output for path /var/lib/kubelet/pods/-29d0-11e6-bafc- /var/lib/kubelet/pods/-29d0-11e6-bafc-  Very strange thing here is when SychPod check the mount path, the last letter "7" of the path is missing. Now the step (4) show that the volume is mounted again (4) 21:20:07. ] Mount command: nsenter [--mount=/rootfs/proc/1/ns/mnt -- /bin/mount -t tmpfs tmpfs /var/lib/kubelet/pods/-29d0-11e6-bafc-] From this point, each time SyncPod check the volume, it mounts again the again which is proved by the findmnt output in the log. After a few hours running, there are over hundreds of mounts.

I am working on finding the reason why the last letter of the returned mount point is missing. I manually tested the findmnt command inside of the container and it returned the correct result. There is also an issue for umount. The umount code will first move the mount dir to a temporary dir, but failed with "device or resource busy" error. I also tried to mount tmpfs in the container with command mount -t tmpfs tmpfs userdir Then tried to mv userdir also failed with the same error. Will continue working on solving this issue. Please let me know if you any comments
Excellent debugging Looking forward to seeing what the root cause is for both issues.
Some updates for this bug issue one that path name missing the last character: I tried to run the "findmnt" command directly on terminal, the output is fine. But when executing the command though exec.command.Output(), the target path name's last character always misses. I tried out different findmnt options, it does not work. Only if the mounting path name is relatively short (looks like under 90~100 characters), there is no missing character. Also no matter how long the path is, only the last character is missing. Our current code in function nsenter_mount.IsLikelyNotMountPoint() uses exact string match to detect whether the path is already mounted or not, which is not robust. We can change to substring match instead. umount issue. The tearDown also invokes IsLikelyNotMountPoint(). If it returns error message, GetMountMedium() function will regard the mount medium as default, but instead it should be memory. Then teaddown will invokes tearDownDefault and tries to rename directories and delete them, which in turn results in "Device or resource busy" error. Also it does not handle the case if path is mounted multiple times (this case should not happen in normal situation) I will try to modify the IsLikeLyNotMountPoint() to make it more robust. Please let me know if you have any comments or suggestions. Thanks!
PR and PR both modified the behavior of to fix issues and evidently introduced different issues. So watch out for regressions in any potential fix. CC Also, have you observed the missing character issue in a non-hyperkube environment, or is there something unique about running locally inside a docker container that causes this to happen?

I see isLikelyNotMountPoint() function is implmented in two places: and My understanding is nsenterMounter is only used when kubelet is running inside of container. nscentermount uses findmnt command result to check the mount point. In , it uses a different approach: It checks whether the directory has a different device as parent, then it is a mountpoint I also wrote a small go program to test the "findmnt" command in my local environment, it also truncate the last letter if the mounting path is relatively long. So this "findmnt" behavior is not related to hyperkube environment I think.
Thanks for the clarification
Thanks for debugging this. May I set the milestone, as I think we should fix this before release?
Sure. THanks! Jing , Lucas Käldström wrote:
Please note that findmnt command does not exist on docker host vm for virtualbox/windows setup (i.e. 'boot2docker'). I'm pretty sure this will be the case for Mac too. The errors that I get are: ] Failed findmnt command for path /var/lib/kubelet/pods/-31ba-11e6-bdbe- exit status 1 and these don't seem to get cleaned up
This problem is still there. Right now I'm observing it on my GKE instances (managed VM, Kubernetes 1.3.5): I noticed it after CPU load jumped from typical 15% to crazy 100% on both my nodes: !
Thank you for providing this information. After checking the log offline, the problem should be fixed in v1.3.6. Please upgrade to v1.3.6 and let us know if you have any problem. Thanks!


Question: } node_count="${NUM_MINIONS}" <del> next_node="10.244.0.0" </del> <ins> next_node="${KUBE_GCE_CLUSTER_CLASS_B:-10.244}.0.0" </ins> node_subnet_size=24 node_subnet_count=$((2 ** (32-$node_subnet_size))) subnets=()

This is an offshoot of . cc Here's a log from a single showing that it's getting it's context screwed over by the other clusters coming up at the same time, even those its local environment never changed: Here's the relevant part of the new script: ( is in a separate commit and allows the override of on the GCE so that this script can work.)
I think this can be resolved by having kube- set the context by exporting KUBECONFIG, so a subsequent run doesn't overwrite it.
After some digging with it's a bit trickier than I thought. If we want to be able to run kube-up in parallel from the same branch/working directory, needs to create a separate kubeconfig for each run. But we also want kube-up to set config such that you can immediately run against the created cluster. Probably best to defer until is resolved.
cc And the script is sitting in


Question: _, err = migration.Migrate(currentInstalledCoreDNSversion, kubeadmconstants.CoreDNSVersion, corefile, false) if err != nil { <del> return err </del> <ins> return errors.Wrap(err, "CoreDNS will not be upgraded") </ins> } return nil }

What happened: Upgrade using kubeadm failed because of CoreDNS migration enchancement What you expected to happen: Upgrade passed correctly How to reproduce it (as minimally and precisely as possible): any 1.15.x Kubernetes with CoreDNS that deployed with any unusual image (self built for example). We using commits counter as part of image tag, e.g - 1.6.2-100. to upgrade to 1.16.x with kubeadm. Anything else we need to know?: Logs part without ignoring preflight: Logs part with ignoring preflight: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g: ):Kernel (e.g. ):Install tools: Kubeadm
/sig cluster-lifecycle
The error came from It seems you should mention this issue on coredns project
True. Will notify them also.
/cc
/assign In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped. wdyt? /cc
Yes. IMO, the warnings should be phrased like "CoreDNS configuration migration will not occur: reason"... so that a user expects the config migration will not be done.
/area kubeadm
SGTM
Upgrade passed correctly I think the best we can do here is to allow the upgrade to pass, but this will, in most practical cases, result in a cluster without a functioning DNS. The most common practical use-case for using a self built image of CoreDNS is to use external plugins. During the kubeadm upgrade, CoreDNS will be replaced with a stock release version of CoreDNS. When that new version of CoreDNS starts, it will not be able to load the existing Corefile's external plugins, and exit. ... or, we could finish the upgrade, leaving the existing CoreDNS deployment untouched, which I think works better.
Yea, it's kinda strong words that "Upgrade passed correctly". Probably, I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.
Yes, I think that would work best...


Question: // Use instance metadata service where possible UseInstanceMetadata bool `json:"useInstanceMetadata" yaml:"useInstanceMetadata"` <del> // Use managed service identity for the virtual machine to access Azure ARM APIs UseManagedIdentityExtension bool `json:"useManagedIdentityExtension"` </del> // Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer <del> MaximumLoadBalancerRuleCount int `json:"maximumLoadBalancerRuleCount"` </del> <ins> MaximumLoadBalancerRuleCount int `json:"maximumLoadBalancerRuleCount" yaml:"maximumLoadBalancerRuleCount"` </ins> } // Cloud holds the config and clients


Question: if len(reconstructedVolumes) > 0 { // Add the volumes to ASW <del> rc.updateStatesNew(reconstructedVolumes) </del> <ins> rc.updateStates(reconstructedVolumes) </ins> // The reconstructed volumes are mounted, hence a previous kubelet must have already put it into node.status.volumesInUse. // Remember to update DSW with this information.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: import ( "errors" "fmt" <ins> "os" </ins> "path" "strconv" "strings"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: i use the standalone-cinder-provisioner and the backend storage is iscsi,the pv and volume can dynamic created,but the device can not attached the k8s minion the standalone-cinder-provisioner pod yaml use the web and the pod running: kube-system standalone-cinder-provisioner--5x6x6 1/1 Running 7 5d5h [root kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc--3140-11e9-bde3- 1Gi RWO Delete Bound kube-system/keystone-sc-pvc standard 6d4h the pv infomation: [root kubectl get pv pvc--3140-11e9-bde3- -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: cinderVolumeId: -0bc7-4e8a-879d- standaloneCinderProvisionerIdentity: creationTimestamp: "2019-02-15T16:38:33Z" finalizers: external- name: pvc--3140-11e9-bde3- resourceVersion: "" selfLink: /api/v1/persistentvolumes/pvc--3140-11e9-bde3- uid: -3140-11e9-bde3- spec: accessModes: ReadWriteOnce capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: keystone-sc-pvc namespace: kube-system resourceVersion: "" uid: -3140-11e9-bde3- iscsi: initiatorName: iqn.2018- iqn: iqn.2099- iscsiInterface: default lun: 1 targetPortal: 19.19.19.

252:3260 persistentVolumeReclaimPolicy: Delete storageClassName: standard volumeMode: Filesystem status: phase: Bound [root standalone]# kubectl get po --all-namespaces -w NAMESPACE NAME READY STATUS RESTARTS AGE default iscsi-standalone 0/1 ContainerCreating 0 2s [root standalone]# kubectl describe pod iscsi-standalone Name: iscsi-standalone Namespace: default Priority: 0 PriorityClassName: <noneNode: 12.12.12.30/12.12.12.30 Start Time: Tue, 26 Feb 2019 17:15:19 +0800 Labels: <noneAnnotations: <noneStatus: Pending IP: Containers: iscsi: Container ID: Image: Image ID: Port: <noneHost Port: <noneState: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <noneMounts: from default-token-frzl8 (ro) /var/www/html from mypd (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: mypd: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: keystone-sc-pvc-storage13 ReadOnly: false default-token-frzl8: Type: Secret (a volume populated by a Secret) SecretName: default-token-frzl8 Optional: false QoS Class: BestEffort Node-Selectors: <noneTolerations: for 300s for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m48s default-scheduler Successfully assigned default/iscsi-standalone to 12.12.12.30 Normal SuccessfulAttachVolume 4m48s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc--39a5-11e9-b3a9-" Warning FailedMount 30s (x10 over 4m40s) kubelet, 12.12.12.30 MountVolume.WaitForAttach failed for volume "pvc--39a5-11e9-b3a9-" : open /sys/class/iscsihost: no such file or directory Warning FailedMount 29s (x2 over 2m45s) kubelet, 12.12.12.

30 Unable to mount volumes for pod "iscsi-standalone_default(-39a7-11e9-bb03-)": timeout expired waiting for volumes to attach or mount for pod "default"/"iscsi-standalone". list of unmounted volumes=[mypd]. list of unattached volumes=[mypd default-token-frzl8] What you expected to happen: the pod can running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:k8s v1.13 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
the k8s not connect the iscsi,so the /sys/class/iscsihost dir not exist,then attachdisk function : // Build a map of SCSI hosts for each target portal. We will need this to // issue the bus rescans. portalHostMap, err := b.deviceUtil.GetISCSIPortalHostMapForTarget(b.Iqn) if err != nil { return "", err } the GetISCSIPortalHostMapForTarget will return err then attach failed,why set the func to the login above?or the /sys/class/iscsihost dir when exist or how can i let the dir appear before run the GetISCSIPortalHostMapForTarget function? thanks
/sig storage
/assign bswartz
/assign
can you review this issue ?thanks
I see a bug here. If the iscsi modules haven't even been loaded yet, the /sys/class/iscsi_host directory won't exist. In this case it's better to return an empty map rather than an error.
yes, if the iscsi can not Initializationthe dir will not exist,maybe return empty map rather than an error or delete the func GetISCSIPortalHostMapForTarget i have commit one pr to resove the problem,can you review it?thanks


Question: } type jobInitialStatus struct { <del> active    int succeed   int failed    int startTime *time.Time </del> <ins> active     int succeed    int failed     int startTime  *time.Time conditions []batch.JobCondition </ins> } func TestControllerSyncJob(t *testing.T) {

When the FailureTarget=False condition is manually, then the Job terminates. The condition should be ignored by the Job controller. the like this: the following script to repro: Issue: the Job gets terminated with the status like this: No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign Thanks!
/sig apps


Question: import ( "fmt" <del> podutil "k8s.io/kubernetes/pkg/api/v1/pod" </del> "time" v1 "k8s.io/api/core/v1"

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: cmd.Flags().Var(&options.token, clientcmd.FlagBearerToken, clientcmd.FlagBearerToken+" for the user entry in kubeconfig") cmd.Flags().Var(&options.username, clientcmd.FlagUsername, clientcmd.FlagUsername+" for the user entry in kubeconfig") cmd.Flags().Var(&options.password, clientcmd.FlagPassword, clientcmd.FlagPassword+" for the user entry in kubeconfig") <del> cmd.Flags().Var(&options.embedCertData, clientcmd.FlagEmbedCerts, "embed client cert/key for the user entry in kubeconfig") </del> <ins> f := cmd.Flags().VarPF(&options.embedCertData, clientcmd.FlagEmbedCerts, "", "embed client cert/key for the user entry in kubeconfig") f.NoOptDefVal = "true" </ins> return cmd }

It acts like a boolean flag but requires an argument, unlike boolean flag is pretty much every environment. We need to find a better way to express the things that this is expressing. Maybe a add a tristate to the pflag lib that behaves the same way but doesn't demand an argument?
cc
The handles "true", "false", "unset" and has a default value. I seem to recall you were looking at changing it, but we needed to preserve the value. The complain here is that you have to do instead of just . /cc
fixing is really easy....


Question: // Validate ensures that the specified values fall within the range of the strategy.  Validation // of this will pass if either the UID is not set, assuming that the image will provided the UID <del> // or if the UID is set it is not root.  In order to work properly this assumes that the kubelet // performs a final check on runAsUser or the image UID when runAsUser is nil. </del> <ins> // or if the UID is set it is not root.  Validation will fail if RunAsNonRoot is set to false. // In order to work properly this assumes that the kubelet performs a final check on runAsUser // or the image UID when runAsUser is nil. </ins> func (s *nonRoot) Validate(pod *api.Pod, container *api.Container) field.ErrorList { allErrs := field.ErrorList{} securityContextPath := field.NewPath("securityContext")

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): PodSecurityPolicy RunAsNonRoot Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta, 1.8.0 (HEAD) Environment: Enable PodSecurityPolicy Cloud provider or hardware configuration: gce, probably any OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: What happened: The PSP admission controller ignores the specified RunAsNonRoot, and overwrites it. What you expected to happen: the admission controller is not supposed to overwrite user specified fields in the podsecuritycontext and securitycontext. How to reproduce it (as minimally and precisely as possible): enable psp, and create a pod with [only] RunAsNonRoot=false specified in a securitycontext. Anything else we need to know:
/sig auth
there are actually 2 bugs here. fails to properly restrict this field when matching against a PSP, which is a latent bug as long as we overwrite the value anyway. overwrites the user-supplied setting for the securitycontext/podsecuritycontext. I suspect the best fix will be to use the API's deep copy function. We are also doing a couple other sketchy things: assigning the "copy" built-in as a variable, and creating an "sc" shadow variable.

I have a half fix in but we should probably fix it in a separate PR, because we may want to patch 1.6.
the effect is that PSP can overly restrict a pod that wanted to run as root, but expressed that as instead of since the workaround is easy (set in your pod if you want to run as root), I don't anticipate backporting the fix.
SGTM. I'll try and get the unit tests in after the GKE summit (on friday).


Question: local message=$1 local match=$2 <del> if echo "$message" | grep -q "$match"; then </del> <ins> if grep -q "${match}" <<< "${message}"; then </ins> echo "Successful" echo "message:$message" echo "has:$match"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug for lack of better idea hack/make-rules/test-cmd-, line 1443 is failing, even when the debug output shows the expected text: from the test log: What happened: See for example https://k8s-
This should be owned by whoever owns the "test-cmd runkubectlget_tests". This doesn't look like a failure in the testing framework or the infrastructure. I believe this is
the linked test is actually failing on the next test, not the .missing test... from the test log: this is very odd:
the issue appears to be with
potential fix in
seems like the same issue for runpluginstests:
same fix will hopefully resolve the issue


Question: // DeltaFIFO's Pop(), Get(), and GetByKey() methods return // interface{} to satisfy the Store/Queue interfaces, but they // will always return an object of type Deltas. List() returns <del> // the newest objects currently in the FIFO. </del> <ins> // the newest object from each accumulator in the FIFO. </ins> // // A DeltaFIFO's knownObjects KeyListerGetter provides the abilities // to list Store keys and to get objects by Store key.  The objects in

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing another PR, I noticed incorrect commentary and code. The primary offense is at , where a comment asserts (incorrectly) that a key can be present in but not and the code removes a key from without ensuring it is absent from . The only saving grace is that this code is actually unreachable (as also noted in the comment). I think it would be better to fix the comment and in this unreachable situation, than to violate an invariant that is critical to other code in this file. also contemplates the possibility that a key can be in and absent from . What you expected to happen: I expected there to be no comments or code that contemplate violations of the invariant at How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): current branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Sorry, I should have noticed this in my earlier pass over this code.
Agreed. Was discussing this with at
We discussed this one, Mike will send PR, Daniel will review.


Question: "sync" "k8s.io/api/core/v1" <del> "k8s.io/apimachinery/pkg/util/sets" </del> utilnode "k8s.io/kubernetes/pkg/util/node" "github.com/golang/glog"

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: dir := path.Join(sys_path, name, "block") if dev, err := io.ReadDir(dir); err == nil { found := false <ins> devName := dev[0].Name() </ins> for _, diskName := range azureDisks { <del> glog.V(12).Infof("azure disk - validating disk %q with sys disk %q", dev[0].Name(), diskName) if string(dev[0].Name()) == diskName { </del> <ins> glog.V(12).Infof("azureDisk - validating disk %q with sys disk %q", devName, diskName) if devName == diskName { </ins> found = true break } } if !found { <del> return "/dev/" + dev[0].Name(), nil </del> <ins> devLinkPaths := []string{"/dev/disk/azure/scsi1/", "/dev/disk/by-id/"} for _, devLinkPath := range devLinkPaths { diskPath, err := getDiskLinkByDevName(io, devLinkPath, devName) if err == nil { glog.V(4).Infof("azureDisk - found %s by %s under %s", diskPath, devName, devLinkPath) return diskPath, nil } glog.Warningf("azureDisk - getDiskLinkByDevName by %s under %s failed, error: %v", devName, devLinkPath, err) } return "/dev/" + devName, nil </ins> } } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a customer , customer is unable to use azure disk in StatefulSet since /dev/sd changed after detach/attach disk. Currently we are using to reference the data disk which could be changed in some condition, see code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): 5 StatefulSet with 1 replica and 1 azure disk mount each 1 StatefulSet and recreate the same StatefulSet would be azure disks could not be usable since /dev/sd changed Update: It seems that on azure china, this issue could repro very easily. Anything else we need to know?: I have checked the official doc: Environment: Kubernetes version (use ): from v1.6.0 to v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I am working on a fix for this issue, would use instead of
We changed how we find disks on nodes quite some time back. What server version is this?
yes, I have already checked the code, the point is we are still using (code is ) to "mount -bind" k8s path, while could be changed when VM is attach/detaching data disks, see And I have also checked related AWS, GCE code, they are using other than ,And I have also checked related AWS, GCE code, they are using other than , see This issue could be stably repro on Azure China, while I could not repro that on other region up to now. I have already got a fix for this, still in testing now.
also add for awareness, I am working on a fix to use other than .
I would not use "/dev/disk/by-id" instead utilize the special UDEV rules we have created for our VMs -- They have the advantage that they create symlinks for all of the disks attached to the VM and the LUN assigned to a disk remains the same. The sysmlinks you find under /dev/disk/azure. For a master-node you will see a layout like the following.

In scsi1 you find the additonal (data) disks A newly assigned disk is goign to appear as lun1,lun2,... Since we are not able to guaranty the order in which disks get assigned to a VM I would go and base the code on this already created layout
thanks for the check. So do you mean we use , instead? actually in Azure VM with UDEV rules created, they have no difference, they link to the same /dev/sd* (see below layout). While I also got some customer cases(e.g. coreos, see ), they don't have such UDEV rules created, which means does not exist, I would say would be more common, every linux distro have that path. From debugging perspective, the path is more intuitive.
BTW, I have a azure disk debugging doc: which uses , it's really intuitive for debugging perpurse.
it is right that at the end we point to /dev/sdi like in the sample above the advantage we have with the lun0 is that it acts as an abstraction layer where we can be sure that an application is always utilizing the disk with lun0, independent what actuall device name gets assigned to it. So at the moment it is ../../../sdi and after a reboot it could become ../../../sdk. But thanks to these UDEV rules /dev/disk/azure/scsi1/lun0 will always point to the right device behind it. I also would favour the usage of the lun over the disk-id if from a maintanance/support point of view as we can directly correlate what lun belongs to what disk. Otherwise I have at first fo to the VM directly to find out what disk might have an issue Regarding the centos issue, if we have the waagent installed on it then those new UDEV rules should be available as well. So it wonders me that we do not have them working well. At least I had not seen any customer SR who had reported an issue with those rules specifically on CentOS
It's not centos, it's coreos issue, see If we use , then we would heavily depends on UDEV rules.

I agree that one depends on these special UDEV rules in this case , but on each Azure VM the agent is to be installed anyway, so that is background why I suggested to use the LUNs
I think proposal is the cleanest and safest. However we will have to tell all Azure users that they have to have azure UDEV rules configured in nodes, even if WAAGENT is not installed.
I did hit the issue, using an ACS based Kubernetes cluster running 1.7.7. We are already in the process of moving over to AKS based clusters, so was simply hoping that there would be some escalated effort to bring 1.8.7 into AKS given that the azure-disk provisioner is the default StorageClass.Not sure if you are looking for any more debug info at this time, but I still have the affected pod active and should be able to get you some more information, so just let me know. that's great, I would like to take a look to make sure it's that issue, could you follow this to collect the data.
Sadly the node became NotReady, and the cluster redeployed the pod to a separate node, which has rectified the issue. We have been seeing the issue with far frequency, so I will make sure to gather the data next time it pops up.
That said, the reason I was confident it was this issue previously is that when I exec'd into the shell, I got the "Input / Output error" upon trying to "ls" the mount point. Furthermore, attempting to re-mount that point gave an error to the extent of "Special device /dev/sde does not exist"
Could you recall the VM had restarted before you hit that disk issue? Or just after attach/detach disk, that issue come up?


Question: ContainerRuntimeOptions: config.ContainerRuntimeOptions{ ContainerRuntime: kubetypes.RemoteContainerRuntime, }, <del> NodeLabels: tt.labels, </del> <ins> RemoteRuntimeEndpoint: "unix:///run/containerd/containerd.sock", NodeLabels:            tt.labels, </ins> }) if tt.error && err == nil {

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.


Question: clusterNameKey = "kubernetes-cluster-name" ) <del> // GetLoadBalancer returns whether the specified load balancer exists, and </del> <ins> // GetLoadBalancer returns whether the specified load balancer and its components exist, and </ins> // if so, what its status is. func (az *Cloud) GetLoadBalancer(ctx context.Context, clusterName string, service *v1.Service) (status *v1.LoadBalancerStatus, exists bool, err error) { <del> _, status, exists, err = az.getServiceLoadBalancer(service, clusterName, nil, false) </del> <ins> // Since public IP is not a part of the load balancer on Azure, // there is a chance that we could orphan public IP resources while we delete the load blanacer (kubernetes/kubernetes#80571). // We need to make sure the existence of the load balancer depends on the load balancer resource and public IP resource on Azure. existsPip := func() bool { pipName, _, err := az.determinePublicIPName(clusterName, service) if err != nil { return false } pipResourceGroup := az.getPublicIPAddressResourceGroup(service) _, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil { return false } return existsPip }() _, status, existsLb, err := az.getServiceLoadBalancer(service, clusterName, nil, false) </ins> if err != nil { <del> return nil, false, err </del> <ins> return nil, existsPip, err </ins> } <del> if !exists { </del> <ins> // Return exists = false only if the load balancer and the public IP are not found on Azure if !existsLb && !existsPip { </ins> serviceName := getServiceName(service) klog.V(5).Infof("getloadbalancer (cluster:%s) (service:%s) - doesn't exist", clusterName, serviceName) return nil, false, nil } <ins> // Return exists = true if either the load balancer or the public IP (or both) exists </ins> return status, true, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The issue has been observed on Azure but should also apply to other providers as well. On Azure we observed that there are orphaned Public IPs that don't belong to any Kubernetes LoadBalancer service but from looking at the tags and the name it is clear that Kubernetes created them. In the Azure Activity Log, you can see that the Public IP was deleted when the K8s service was deleted but recreated a few minutes later. That re-creation is also visible in the K8s controller manager logs as a reconcile of the service. In our case, we add additional worker nodes and delete services at the same time. I think this leads to the re-creation here Because the service list is retrieved at the beginning of the reconcile loop, services that are deleted while the loop is reconciling a recreated by that loop and afterwards orphaned. What you expected to happen: No orphaned public IPs How to reproduce it (as minimally and precisely as possible): Create a few Kubernetes Load Balancer services (we have around 20). Then scale your worker nodes and delete one service at the same time. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: Azure OS (e.g: ): Core OS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
Thanks for reporting the issue. I think this should be fixed during v1.16 release cycle. /milestone v1.16
In the nodeSyncLoop -updateLoadBalancerHosts -lockedUpdateLoadBalancerHosts call chain, I don't see where the mutex lock is obtained.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 4 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged.

Is this issue still planned to be fixed for the 1.16 release?
The issue would probably not able to be fixed in v1.16. AFAIK, no one is working on it now.
Thank you for the update! I'll go ahead and remove the issue from the milestone. If anything changes in the meanwhile, please let us know. /milestone clear
/assign
/assign
GitHub didn't allow me to assign the following users: chewong. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't LB Finalizer support fix this? We enabled this by default starting v1.16
Initial PR: Bump to beta in v1.16:


Question: flags := NewEventsFlags(restClientGetter, streams) cmd := &cobra.Command{ <del> Use:                   fmt.Sprintf("events [(-o|--output=)%s] [--for TYPE/NAME] [--watch] [--event=Normal,Warning]", strings.Join(flags.PrintFlags.AllowedFormats(), "|")), </del> <ins> Use:                   fmt.Sprintf("events [(-o|--output=)%s] [--for TYPE/NAME] [--watch] [--types=Normal,Warning]", strings.Join(flags.PrintFlags.AllowedFormats(), "|")), </ins> DisableFlagsInUseLine: true, Short:                 i18n.T("List events"), Long:                  eventsLong,

The seems to be wrong, should be instead of No response <details</details<detailsI am running local Kubernetes cluster using kind. </details<details</details<detailsDocker Desktop for Mac Kind Kubernetes Cluster </details<detailsDocker </details<detailskindnet </details
/sig cli
/triage accepted /priority backlog would you like to fix this mistyping?
Yes Sir! Abid
Hello, can i work on this as my first contribution pls ?
Actually this will be first for me also
please assign yourself and work on it
/assign
/assign
I will assign it to myself once I get home, I am away right now. Thanks
- How unethical it is that you are assigning an issue that you have not pointed out. It is much better that you find a issue and repot it and then assign it. Please go ahead and unassign it, I will assign it and will work on it. CC
/assign
As you opened issue and said that you are planning to work on this, you fully deserve to work on that issue. I'm sorry that I did not check the assignee and approved the PR fixing this. next time, I highly suggest that check if there is assignee or not and only start work on it, only if there is noone.
Oh sorry I thought you said 'please assign yourself and work on it'. I will cancel my PR. I am really sorry.
I was on my phone and i received a mail, so I though it was you who said that ... I cancelled the PR.


Question: // Tests can be run in parallel, so we need a different nodePort for // each test. <del> // We add 1 to FederatedSvcNodePortLast because IntnRange's range end // is not inclusive. nodePort := int32(rand.IntnRange(FederatedSvcNodePortFirst, FederatedSvcNodePortLast+1)) </del> <ins> // we add in a array all the "available" ports availablePorts := make([]int32, FederatedSvcNodePortLast-FederatedSvcNodePortFirst) for i := range availablePorts { availablePorts[i] = int32(FederatedSvcNodePortFirst + i) } var err error var service *v1.Service retry := 10 // the function should retry the service creation on different port only 10 time. // until the availablePort list is not empty, lets try to create the service for len(availablePorts) > 0 && retry > 0 { // select the Id of an available port i := rand.Intn(len(availablePorts)) By(fmt.Sprintf("try creating federated service %q in namespace %q with nodePort %d", name, namespace, availablePorts[i])) service, err = createServiceWithNodePort(clientset, namespace, name, availablePorts[i]) if err == nil { // check if service have been created properly in all clusters. // if the service is not present in one of the clusters, we should cleanup all services if err = checkServicesCreation(namespace, name, clusters); err == nil { // everything was created properly so returns the federated service. return service, nil } } // in case of error, cleanup everything if service != nil { if err = deleteService(clientset, namespace, name, nil); err != nil { framework.ExpectNoError(err, "Deleting service %q after a partial createService() error", service.Name) return nil, err } cleanupServiceShardsAndProviderResources(namespace, service, clusters) } // creation failed, lets try with another port // first remove from the availablePorts the port with which the creation failed availablePorts = append(availablePorts[:i], availablePorts[i+1:]...) retry-- } return nil, err } func createServiceWithNodePort(clientset *fedclientset.Clientset, namespace, name string, nodePort int32) (*v1.Service, error) { </ins> service := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      name,

In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


Question: cluster/cloudcfg.sh list /pods ``` <del> You should see a single redis master pod.  It will also display the machine that the pod is running on.  If you ssh to that machine, you can run </del> <ins> You'll see a single redis master pod. It will also display the machine that the pod is running on. ```javascript Using master: kubernetes-master (external IP: 1.2.3.4) { "kind": "cluster#podList", "items": [ { "id": "redis-master-2", "labels": { "name": "redis-master" }, "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "name": "master", "image": "dockerfile/redis", "ports": [ { "hostPort": 6379, "containerPort": 6379 } ] } ] } }, "currentState": { "manifest": { "version": "", "volumes": null, "containers": null }, "host": "kubernetes-minion-3.c.{project-id}.internal" } } ] } ``` If you ssh to that machine, you can run `docker ps` to see the actual pod: </ins> ```shell <del> sudo docker ps </del> <ins> $ gcloud compute ssh kubernetes-minion-3 --zone us-central1-b $ sudo docker ps me@kubernetes-minion-3:~$ sudo docker ps CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES 417ab993cdf8  dockerfile/redis:latest  redis-server /etc/re  8 minutes ago Up 8 minutes  0.0.0.0:6379->6379/tcp  master--redis_-_master_-_2--6b944b49 </ins> ``` <del> And see the actual pod.  (Note that initial `docker pull` may take a few minutes, depending on network conditions.) </del> <ins> (Note that initial `docker pull` may take a few minutes, depending on network conditions.) </ins> ### Step Two: Turn up the master service. <del> A Kubernetes 'service' is a named load balancer that proxies traffic to one or more containers.  The services in a Kubernetes cluster are discoverable inside other containers via environment variables.  Services find the containers to load balance based on pod labels.  The pod that you created in Step One has the label `name=redis-master`, so the corresponding service is defined by that label.  Create a file named `redis-master-service.json` that contains: </del> <ins> A Kubernetes 'service' is a named load balancer that proxies traffic to one or more containers. The services in a Kubernetes cluster are discoverable inside other containers via environment variables. Services find the containers to load balance based on pod labels. </ins> <del> ```javascript </del> <ins> The pod that you created in Step One has the label `name=redis-master`, so the corresponding service is defined by that label.  Create a file named `redis-master-service.json` that contains: ```js </ins> { "id": "redismaster", "port": 10000,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: # Kubectl diff # ################ record_command run_kubectl_diff_tests <ins> record_command run_kubectl_diff_same_names </ins> ############### # Kubectl get #

What happened: I get a diff, when I specify a file directly, but on recursive mode, no diff is shown anymore: What you expected to happen: Recursive mode should work How to reproduce it (as minimally and precisely as possible): Have a bunch of YAML manifest, but with name collisions, e.g. Service , Deployment , etc Anything else we need to know?: I think the problem is that all objects have the same name, which should not be a problem per se. Only heapster-binding has a different name, and looking at the generated temp files, only and are present: Environment: Kubernetes version (use ): $ kubectl version --short Client Version: v1.13.0 Server Version: v1.13.0 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
I think the files are overwriting each other?
Yeah, good catch, I'll try to fix that asap
Thanks! By the way, very nice work with dry-run and diff. I was waiting for this.
Feel free to look at the PR and give your opinion!


Question: triggerFunc storage.TriggerPublisherFunc) (storage.Interface, factory.DestroyFunc) { s, d := generic.NewRawStorage(storageConfig) <del> if capacity == 0 { </del> <ins> if capacity <= 0 { </ins> klog.V(5).Infof("Storage caching is disabled for %T", objectType) return s, d }

Currently we signal undecorated storage by simply affecting the undecoratedStorage to the decorator. This is a blocker for kubernetes/enhancements as we need to enable watch cache for events (which currently hardcoded and not configurable) cc /assign /sig api-machinery


Question: Name:  "remove-etcd-member", Short: "Remove a local etcd member.", Long:  "Remove a local etcd member for a control plane node.", <del> Run:   runPreflight, </del> <ins> Run:   runRemoveETCDMemberPhase, </ins> InheritFlags: []string{ options.KubeconfigPath, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running fails to remove etcd member. What you expected to happen: I expect to remove the node from the etcd cluster. How to reproduce it (as minimally and precisely as possible): From a working 1.15.0 cluster, run on a control plane node. Anything else we need to know?: I think the issues is due to the wrong function being used here: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /cc
/area kubeadm
/sig cluster-lifecycle