Question: observations, _ := makeSignalObservations(summary) debugLogObservations("observations after resource reclaim", observations) <del> // determine the set of thresholds met independent of grace period thresholds := thresholdsMet(m.config.Thresholds, observations, false) </del> <ins> // evaluate all thresholds independently of their grace period to see if with // the new observations, we think we have met min reclaim goals thresholds := thresholdsMet(m.config.Thresholds, observations, true) </ins> debugLogThresholdsWithObservation("thresholds after resource reclaim - ignoring grace period", thresholds, observations) if len(thresholds) == 0 {

When both and parameters are set on Kubelet, they act as the upper and lower thresholds that define how the state is entered and exited. When trying to remediate a , Kubelet performs different operations to free up space. In order to decide if it has freed up enough, it compares the currently available disk space with the triggering threshold instead of the exiting threshold. Kubelet ends up in a long lasting state which can only be left with Kubelet restart, or if by luck the Pods that were using disk space are rescheduled somewhere else. Kubelet periodically does some housekeeping in order to ensure its node has enough resources available. If that isn’t the case, it tries to free up resources, updates the node’s condition (e.g. ) and in last resort starts evicting Pods. By default, the housekeeping is done every 10 seconds for a set of “signals” which represent node level resources (e.g. , , , ). There are a couple of parameters that comes into play in this process and I'll ignore all but two for simplicity: Signal thresholds: defines at which point there is not enough of a resource available, e.g. Signal minimum reclaim: how much resources Kubelet should aim to free up before a threshold is considered as not met. It’s the lower bound of an hysteresis, the threshold being the upper bound (e.g. ) The housekeeping is implemented in Here is my understanding of what this method does, filtering out what I think does matter in this issue: the signals (e.g. how much is available) a list of thresholds that are met (e.g. observed < ) there were thresholds met on a previous run: a. Check if those previous threshold are still met taking the into account. If = 15% and , our value to unmet the threshold is actually 25% percent b. Merge them with (2), the thresholds that are met from the new observation (1) out thresholds for which the individual grace period is not reached. They might be taken into account in another run if observed again. the previously met threshold with the new computed list of thresholds. Those met threshold are stored across runs. no threshold is not actually met, exits. If at least one is met, Kubelet has work to do! Try to reclaim node level resources.

In other terms, try to free up resources without having to evict any Pod. For that’s done by removing stopped containers, and unused images. a new observation. If the threshold are not met anymore, ignoring , exit with Pods as last resort Ignoring the in (7) seems wrong. If the amount of resources freed is somewhere between and , Kubelet can get stuck. On the next run, as part of (3a) the threshold taking into account is not unmet. Therefore Kubelet tries again to free up node level resources in (6), eventually does nothing but exits (6) pretending it free up enough resources without doing anything. after having cleaned up the node and in order to decide if Pod eviction is the step, seems to do the trick. Kubelet should try to get out of a by evicting almost all Pods. In this case, it means it should try to evict Pods until the threshold is reached and the can be removed. I expect Kubelet to stay in permanent only if the resources that are used are beyond its control. Preparation: a node with a different from (the problem might be the same when they are mixed but I didn't try) to 15% and to 30% in the Kubelet’s configuration into a Pod running on that node, and use to write a file inside the container. The amount of data should be enough to bring the underlying disk at 86% usage for Kubelet to notice the threshold and to cleanup stopped containers and remove unused images Remove your file. The node is now clean. Restart Kubelet and notice how it sees everything is normal Getting Kubelet stuck: into your Pod and write a file to reach 86% usage again for Kubelet to notice, cleanup stopped containers and images. Since the cleanup was done in the preparation step a minute ago, there is almost nothing to be removed and your usage should stay more or less the same the Pod's file by a new one to reach between 55% and 85% disk usage. how every 10 seconds, Kubelet pretends it was able to . The Node is stuck in until you restart Kubelet and it forgets it had to ensure there is at least available in .

Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: vSphere OS (e.g: ): 18.04.2 LTS Kernel (e.g. ): 4.15.0-130-generic We have been running a slightly modified version of Kubelet on one of our clusters for the past 5 days, and since then we had no node staying in state for more than 20 minutes. From the disk usage graph, it looks like both image garbage collection and the disk pressure remedy operations are working properly. I'll open a PR with the little patch we're using.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: [Kustomize](https://github.com/kubernetes-sigs/kustomize) version in kubectl had a jump from v2.0.3 to [v4.0.5](https://github.com/kubernetes/kubernetes/pull/98946). Kustomize is now treated as a library and future updates will be less sporadic. <del> ### Default Container Labels </del> <ins> ### Default Container Annotation </ins> <del> Pod with multiple containers can use `kubectl.kubernetes.io/default-container` label to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). </del> <ins> Pod with multiple containers can use `kubectl.kubernetes.io/default-container` annotation to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). </ins> ### Immutable Secrets and ConfigMaps

is wrong as it says default label can be specified, but KEP and other lines of changelog mentions annotation. Changelog is correct and consistent about default container feature for kubectl. I'm still not able to get this feature to work actually for . Kubernetes version (use ): v1.21.0
/triage accepted thanks for reporting early!
/remove-kind bug /kind documentation
/sig release


Question: . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework"

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: "errors" "fmt" "sync" <ins> "time" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/klog/v2" <ins> utiltrace "k8s.io/utils/trace" </ins> ) // DeltaFIFOOptions is the configuration parameters for DeltaFIFO. All are

I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?)

I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn


Question: "reflector_test.go", "shared_informer_test.go", "store_test.go", <ins> "thread_safe_store_test.go", </ins> "undelta_store_test.go", ], embed = [":go_default_library"],

What happened: Consider this use case, a k8s cluster with many short lived name spaces with a high cardinality. (Think UUIDs). Over time, some of the indices contain string sets for each name space with no objects in the backing string sets; which are never reused. Leads to process memory increasing over time until the process exceeds available memory. Potentially causing issues as an api-server or controller is terminated by the operating system. What you expected to happen: Process memory does not increase unbounded as the number of unique items then deleted from a thread safe map, causing indices to hold on to zero length string sets for the key. This could be mitigated when an index's string set for a key is empty, that string set is removed from the index. How to reproduce it (as minimally and precisely as possible): Thanks to for this reproduction code: Anything else we need to know?: Can be mitigated with the following patch against master at : has a code path for creating sets that don't exist, so if a set is temporarily reduced to zero; it will be re-created as if it was the initial case where the set was never created. Here's an apiserver pprof heap base png from a clean start for a test case of ≈800k name space create/delete in kube v1.14.7: Note the increase in heap utilisation from base of and nodes in the hot path. Similar increases were observed in v1.13.11 and v1.15.5. Same as above, but with the suggested fix and ≈100k name space create/delete: Note lack of and nodes in the hot path. (Was not able to directly attach the images to this issue due to Something went really wrong, and we can't process that file.) Environment: Kubernetes version (use ): Issue reproduced in v1.13.11, v1.14.7, 1.15.5 directly by adding/deleting namespaces. Above mitigation directly tested against a v1.14.7 cluster.
/sig api-machinery
Edit: I found threadSafeMap.updateIndices() - was searching with misspelled func name.
I'm not sure I follow, but is still present -
Nah, my bad; didn't spell the method right originally in the text.

One option is to mark the time when set becomes empty: We can clean Index entry when certain period has passed since the set becomes empty.
I really don't think that solution is a viable and adds unneeded complexity. You would then need to have something running in the background to prune empty sets. The solution proposed in the original post is the most straight forward, when the set is empty - delete it. If something is then created that needs the set, it will be recreated when needed.
I think that is fine.


Question: for i := 0; i < rss.offset; i++ { at := (start + i) % rss.offset if allocated.Bit(at) == 0 { <del> return i, true </del> <ins> return at, true </ins> } } return 0, false

In a enabled cluster, Kubernetes assigns duplicate IP addresses when the dynamic block is exhausted. I expected all clusterIPs are uniquely assigned. a enabled Kubernetes v1.24 cluster with a small service-cluster-ip-range. I confirmed on the minikube with the following command. services until the dynamic block is exhausted. see some clusterIPs are duplicated. No response <details</details<detailsminikube v1.25.2 </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When the dynamic block is exhausted, we expect the service creation to fail with an error message like "No IP address available", right?
According to , Dynamically IP assignment will use the upper band by default, once this has been exhausted it will use the lower range. So I think using the reserved block (the lower range) is expected behavior
/sig network
upps /assign
/assign great catch, thanks


Question: }, }, }, <ins> InitContainers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: requests, Limits:   limits, }, }, }, </ins> }, } }

The function removeMissingExtendedResources() located at , is designed to remove any extended resources from a container’s requests that are not found in nodeInfo.Allocatable before the pod is admitted. This is necessary to support cluster-level resources, which are extended resources that are unknown to nodes. However, this function only removes missingExtendedResources from the container’s requests, and does not process the requests of initContainers. This could lead to issues, as initContainers might request resources that do not exist on the node, which could result in the pod failing to be admitted or other unforeseen issues. As such, it may be necessary to refine the removeMissingExtendedResources() function to also handle initContainer’s requests. Refine the removeMissingExtendedResources() function to also handle initContainer’s requests. Create a pod that requests cluster-level extended resources in both the container and InitContainer. No response The problem persists in the latest version. <details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-longterm


Question: return err } // For connection errors and apiserver shutdown errors retry. <del> if net.IsConnectionReset(err) || net.IsConnectionRefused(err) { </del> <ins> if net.IsConnectionReset(err) { </ins> // For the purpose of retry, we set the artificial "retry-after" response. // TODO: Should we clean the original response if it exists? resp = &http.Response{

/kind bug changed behavior of client-go on connection refused errors, which changed a short wait and an "invalid host/port" error into a long retry wait and a confusing "internal server" error /assign Originally posted by in
/sig api-machinery
/milestone v1.18
I linked
Hello, bug-triage here. Just want to make sure we are still staffed and on-track for v1.18 code freeze
thanks, I will get to this tomorrow.
gentle ping


Question: set -e source "${KUBE_ROOT}/hack/lib/init.sh" <ins> kube::util::ensure-gnu-sed </ins> function usage { echo "This script starts a local kube cluster. "

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and


Question: fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeCombinedOutputAction{ func() ([]byte, error) { return []byte("dummy device have been created"), nil }, <ins> func() ([]byte, error) { return []byte(""), nil }, </ins> }, } fexec := &fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, <ins> func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, </ins> }, LookPathFunc: func(cmd string) (string, error) { return cmd, nil }, }

Is this a BUG REPORT or FEATURE REQUEST? /kind bug What happened: In ipvs proxy mode, most service functions like "nodeport/loadbalance" is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional. What you expected to happen: Check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.
/sig network /area ipvs /assign
What's the current UT coverage number?
Current is 50.2%.
So, can we achieve 70%?
Yeah, we can do that, and that's definitely necessary.


Question: } bootstrapClientConfig, err := loadRESTClientConfig(kubeconfigPath) if err != nil { <del> utilruntime.HandleError(fmt.Errorf("unable to read existing bootstrap client config: %v", err)) </del> <ins> utilruntime.HandleError(fmt.Errorf("unable to read existing bootstrap client config from %s: %v", kubeconfigPath, err)) </ins> return false, nil } transportConfig, err := bootstrapClientConfig.TransportConfig() if err != nil { <del> utilruntime.HandleError(fmt.Errorf("unable to load transport configuration from existing bootstrap client config: %v", err)) </del> <ins> utilruntime.HandleError(fmt.Errorf("unable to load transport configuration from existing bootstrap client config read from %s: %v", kubeconfigPath, err)) </ins> return false, nil } // has side effect of populating transport config data fields if _, err := transport.TLSConfigFor(transportConfig); err != nil { <del> utilruntime.HandleError(fmt.Errorf("unable to load TLS configuration from existing bootstrap client config: %v", err)) </del> <ins> utilruntime.HandleError(fmt.Errorf("unable to load TLS configuration from existing bootstrap client config read from %s: %v", kubeconfigPath, err)) </ins> return false, nil } certs, err := certutil.ParseCertsPEM(transportConfig.TLS.CertData) if err != nil { <del> utilruntime.HandleError(fmt.Errorf("unable to load TLS certificates from existing bootstrap client config: %v", err)) </del> <ins> utilruntime.HandleError(fmt.Errorf("unable to load TLS certificates from existing bootstrap client config read from %s: %v", kubeconfigPath, err)) </ins> return false, nil } if len(certs) == 0 { <del> utilruntime.HandleError(fmt.Errorf("unable to read TLS certificates from existing bootstrap client config: %v", err)) </del> <ins> utilruntime.HandleError(fmt.Errorf("unable to read TLS certificates from existing bootstrap client config read from %s: %v", kubeconfigPath, err)) </ins> return false, nil } now := time.Now() for _, cert := range certs { if now.After(cert.NotAfter) { <del> utilruntime.HandleError(fmt.Errorf("part of the existing bootstrap client certificate is expired: %s", cert.NotAfter)) </del> <ins> utilruntime.HandleError(fmt.Errorf("part of the existing bootstrap client certificate in %s is expired: %v", kubeconfigPath, cert.NotAfter)) </ins> return false, nil } }

What happened: the fails to start; it logs, What you expected to happen: either the starts, or it produces a useable diagnostic It tells me that a certificate is expired. If you Google the error, you'll hit SO questions like , which helpfully suggest you run ¹. That runs successfully, but afterwards, the continues to fail to start. A bit more digging, and one might learn about , and think, ah-ha! That'll tell me which cert has expired, and thus, where to look for guidance on renewing it. As we've already run , it will tell us that … …nothing is expired. (But still the refuses to start.) So we just run every in through manually with : …and they're all fine. After some further digging, I did find it. It's in , but it's wrapped in YAML config so the can't see it. It isn't clear to me why should miss it. But at the very least, needs to emit a usable diagnostic, minimally including which* certificate is expired, and ideally with enough information for me to determine the correct way to fix it. (Eventually, I hit , which contains the additional command to regenerate .) How to reproduce it (as minimally and precisely as possible): run a cluster for ~1 yr, let the cert expire… Anything else we need to know?: ¹it instills much confidence that renewing the certs is "alpha". Environment: Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: N/A, single -made node OS (e.g: ): Gentoo Kernel (e.g. ): Install tools: ? Network plugin and version (if this is a network-related bug): N/A, but Calico Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems that the certs are renewed today. You may need to restart kube-apiserverkube-controllerkube-scheduleretcd & kubelet on each node.

It's suggested to run before it expired.
this version is no longer in support. you seem to have sufficient information on how to recover. arguably it does need better messaging. i will keep this open with a "feature" label, but if you want this changed you might have to send the PR yourself. the guide to create certs is here: kubeadm makes it easier for users, but the kubelet error messages should not mention kubeadm. /remove-kind bug /kind feature /area kubelet
/sig node
/retitle kubelet: improve error messages about expired certificates
also leaving this to SIG Node to decide if the want the feature or not. (via "triage accepted" vs "close")
If you don't mind, I sent a PR to improve the error message.


Question: package gce_cloud import ( <ins> "crypto/md5" </ins> "fmt" "io" "io/ioutil"

GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated).

Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.

I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.


Question: if err != nil { return err } <del> err = testReachabilityOverNodePorts(nodes, servicePort, pod, clusterIP) </del> <ins> err = testReachabilityOverNodePorts(nodes, servicePort, pod, clusterIP, j.ExternalIPs) </ins> if err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: "STANDARD_D15_V2":        64, "STANDARD_D16AS_V4":      32, "STANDARD_D16A_V4":       32, <ins> "STANDARD_D16DS_V4":      32, "STANDARD_D16D_V4":       32, </ins> "STANDARD_D16S_V3":       32, "STANDARD_D16_V3":        32, "STANDARD_D1_V2":         4, "STANDARD_D2":            8, "STANDARD_D2AS_V4":       4, "STANDARD_D2A_V4":        4, <ins> "STANDARD_D2DS_V4":       4, "STANDARD_D2D_V4":        4, </ins> "STANDARD_D2S_V3":        4, "STANDARD_D2_V2":         8, "STANDARD_D2_V2_PROMO":   8,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: accBody: "X-Muffins-Or-Cupcakes: muffins", }, { <ins> handler: headerEchoHandler, reqHeaders: http.Header{ "User-Agent": {"foo/1.0"}, }, health:  probe.Success, accBody: "User-Agent: foo/1.0", }, { handler: headerEchoHandler, reqHeaders: http.Header{ "User-Agent": {""}, }, health:  probe.Success, notBody: "User-Agent", }, { handler:    headerEchoHandler, reqHeaders: http.Header{}, health:     probe.Success, accBody:    "User-Agent: kube-probe/", }, { </ins> // Echo handler that returns the contents of Host in the body handler: func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200)

Grabbed tcpdump of a readiness probe - the src ip was the bridge address (10.244.1.1), but apart from this, there is no identifying information in the HTTP request that this is a Kubernetes daemon requesting readiness checks - this might help QoS/priority assignment by the container application.
IIUC, you mean we should add a better when kubernetes daemon request a readiness check to identify this request comes from kubernetes daemon?
Yes, we should override the default net/http user-agent with our own. Better usability for users and we have no future upgrade hazards if users start hardcoding and expecting this default User-Agent string in their analytic tools.
Any news about this one? It would be nice! I've this little hack in my nginx configuration (in order to not be spamed by health check in logs) :)
This can probably be closed now that probes have support.


Question: // NOTE: Anything passed to DockerService should be eventually handled in another way when we switch to running the shim as a different process. func NewDockerService(client dockertools.DockerInterface, seccompProfileRoot string, podSandboxImage string, streamingConfig *streaming.Config, <del> pluginSettings *NetworkPluginSettings, cgroupsName string, kubeCgroupDriver string) (DockerService, error) { </del> <ins> pluginSettings *NetworkPluginSettings, cgroupsName string, kubeCgroupDriver string, execHandler dockertools.ExecHandler) (DockerService, error) { </ins> c := dockertools.NewInstrumentedDockerInterface(client) ds := &dockerService{ seccompProfileRoot: seccompProfileRoot,

Either deprecate the nsenter exec handler, or add nsenter handling to the dockershim (after goes in). /cc
I vote for deprecating the nsenter exec handler for docker. has been the default exec handler since kubernetes pre-1.0. Is there any reason (or use case) for keeping the pluggable exec handler? Another question is that whether we can drop exec in one release, or we need to provide support in CRI for at lease one more release. /cc
+1 for docker exec.
I suggest deprecating docker nsenter exec in a long term, but to achieve that, we need to follow the deprecation policy. The first thing we need to address is marking this flag (--docker-exec-handler, default: native) being deprecated:
Also there are several issues raised before which are related to the issue of docker exec in various versions. The engineer even suggested to switch the default mechanism is docker nsenter exec. One of the examples is
Now let's narrow down to the CRI scope, and below is my 2 cents: 1) Can we mark the flag deprecated in 1.6? 2) Drop the support in CRI in 1.6? Inform the kubernetes users with this decision along CRI announcement being scheduled sometimes in February? 3) Drop the support completely from Kubernetes unless there are real use cases coming up.
We'll have to tell the users that they need to disable CRI in order to use the nsenter exec plugin. could you give a quick estimate about the amount of work required to add support for nsenter exec in dockershim? Thanks.
Talked to offline. I'll Add support nsenter as exec in dockershim Mark the flag as deprecated.


Question: timeOfLastSync                  time.Time volumesFailedReconstruction     []podVolume volumesNeedUpdateFromNodeStatus []v1.UniqueVolumeName <del> volumesNeedReportedInUse        []v1.UniqueVolumeName </del> } func (rc *reconciler) unmountVolumes() {

remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign


Question: ) const ( <del> dockerProcessName     = "docker" </del> <ins> dockerProcessName     = "dockerd" </ins> dockerPidFile         = "/var/run/docker.pid" containerdProcessName = "docker-containerd" containerdPidFile     = "/run/docker/libcontainerd/docker-containerd.pid"

This issue was reported in the Description The container manager used in kubelet checks for docker daemon process either via pidfile or process name. While the pidfile points to the docker daemon process PID, the dockerProcessName constant stores a docker cli name (docker) instead of docker daemon name (dockerd). Figure 30.1: Constants in file. Recommendation Correct the docker process name to dockerd. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-025 and it was finding 31 of the report. The vendor considers this issue Informational Severity. To view the original finding, begin on page 77 of the Environment: Kubernetes version: 1.13.4
/sig node /kind bug /remove-kind feature
/lgtm
lol, this is the issue. my bad!


Question: go_test( name = "go_default_test", <del> srcs = ["defaults_test.go"], </del> <ins> srcs = [ "conversion_test.go", "defaults_test.go", ], </ins> embed = [":go_default_library"], deps = [ "//pkg/api/legacyscheme:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g.

): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all.

I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix?

:)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via


Question: } volume := Volume{ <del> ID:     volumeV3.ID, Name:   volumeV3.Name, Status: volumeV3.Status, Size:   volumeV3.Size, </del> <ins> AvailabilityZone: volumeV3.AvailabilityZone, ID:               volumeV3.ID, Name:             volumeV3.Name, Status:           volumeV3.Status, </ins> } if len(volumeV3.Attachments) > 0 {

Resizing a persistent volume is partially broken on openstack (cinder) with k8s v1.11.1. Partially in the sense that static resizing (requiring a pod restart) works, but the newly introduced online resizing (not requiring a pod restart) fails. Upon resizing a pvc it remains in the Resizing condition forever (see the warning event) until I delete the pod that has this pvc mounted: After pod deletion it notifies me (correctly) that it's now waiting for the pod to come up to finish resizing: So far the warning event does not break the ExpandPersistentVolumes feature, as this one anyhow requires a pod restart. However it breaks the newly introduced ExpandInUsePersistentVolumes feature (alpha with k8s v1.11; doesn't require a pod restart for resizing to finish), as the pvc will remain in "Resizing" condition forever (or until the pod is restarted but this defeats the purpose of ExpandInUsePersistentVolumes). I'm not sure whether this is fully related but upon digging around I found and a consecutive fix with which was merged on Jan 23. However a few days later this fix seems to be (probably accidentally) overwritten with this commit: /kind bug /sig storage /sig openstack
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It is already fixed. So I will close it.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am trying cinder online pvc resizing on kubernetes1.12.2. I noticed this kubernetes version has include your fix, but I still failed to resize pvc online.

The error log has no change: Warning VolumeResizeFailed 45s (x2 over 66s) volumeexpand Error expanding volume "default/myclaim-expend" of plugin : volume status is not available The error log is reported below code line "volume.Status != volumeAvailableStatus" // ExpandVolume expands the size of specific cinder volume (in GiB) func (os *OpenStack) ExpandVolume(volumeID string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) { volume, err := os.getVolume(volumeID) if err != nil { return oldSize, err } if volume.Status != volumeAvailableStatus { // cinder volume can not be expanded if its status is not available return oldSize, fmt.Errorf("volume status is not available") } One point confused me, once cinder volume is used by pod, it's status is "in-use", how the online pvc resizing pass the code check "volume.Status != volumeAvailableStatus"? There's where the online pvc resizing fails.
It looks like Cinder can not support online volume expansion - which is what is happening here. Even if online expansion is enabled, I think to actually perform resizing of Cinder volumes - you must bring the pod down.


Question: TargetContainer string TTY             bool <ins> attachChanged         bool </ins> deprecatedInvocation  bool shareProcessedChanged bool

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: <ins> #!/usr/bin/env bash # Copyright 2020 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail run_kubectl_debug_pod_tests() { set -o nounset set -o errexit create_and_use_new_namespace kube::log::status "Testing kubectl debug (pod tests)" ### Pod Troubleshooting by Copy # Pre-Condition: Pod "nginx" is created kubectl run target "--image=${IMAGE_NGINX:?}" "${kube_flags[@]:?}" kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target:' # Command: create a copy of target with a new debug container kubectl alpha debug target -it --copy-to=target-copy --image=busybox --container=debug-container --attach=false "${kube_flags[@]:?}" # Post-Conditions kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target:target-copy:' kube::test::get_object_assert pod/target-copy '{{range.spec.containers}}{{.name}}:{{end}}' 'target:debug-container:' kube::test::get_object_assert pod/target-copy '{{range.spec.containers}}{{.image}}:{{end}}' "${IMAGE_NGINX:?}:busybox:" # Clean up kubectl delete pod target target-copy "${kube_flags[@]:?}" # Pre-Condition: Pod "nginx" is created kubectl run target "--image=${IMAGE_NGINX:?}" "${kube_flags[@]:?}" kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target:' # Command: create a copy of target with a new debug container replacing the previous pod kubectl alpha debug target -it --copy-to=target-copy --image=busybox --container=debug-container --attach=false --replace "${kube_flags[@]:?}" # Post-Conditions kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target-copy:' kube::test::get_object_assert pod/target-copy '{{range.spec.containers}}{{.name}}:{{end}}' 'target:debug-container:' kube::test::get_object_assert pod/target-copy '{{range.spec.containers}}{{.image}}:{{end}}' "${IMAGE_NGINX:?}:busybox:" # Clean up kubectl delete pod target-copy "${kube_flags[@]:?}" # Pre-Condition: Pod "nginx" is created kubectl run target "--image=${IMAGE_NGINX:?}" "${kube_flags[@]:?}" kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target:' kube::test::get_object_assert pod/target '{{(index .spec.containers 0).name}}' 'target' # Command: copy the pod and replace the image of an existing container kubectl alpha debug target --image=busybox --container=target --copy-to=target-copy "${kube_flags[@]:?}" -- sleep 1m # Post-Conditions kube::test::get_object_assert pod "{{range.items}}{{${id_field:?}}}:{{end}}" 'target:target-copy:' kube::test::get_object_assert pod/target-copy "{{(len .spec.containers)}}:{{${image_field:?}}}" '1:busybox' # Clean up kubectl delete pod target target-copy "${kube_flags[@]:?}" set +o nounset set +o errexit } run_kubectl_debug_node_tests() { set -o nounset set -o errexit create_and_use_new_namespace kube::log::status "Testing kubectl debug (pod tests)" ### Node Troubleshooting by Privileged Container # Pre-Condition: Pod "nginx" is created kube::test::get_object_assert nodes "{{range.items}}{{${id_field:?}}}:{{end}}" '127.0.0.1:' # Command: create a new node debugger pod output_message=$(kubectl alpha debug node/127.0.0.1 --image=busybox --attach=false "${kube_flags[@]:?}" -- true) # Post-Conditions kube::test::get_object_assert pod "{{(len .items)}}" '1' debugger=$(kubectl get pod -o go-template="{{(index .items 0)${id_field:?}}}") kube::test::if_has_string "${output_message:?}" "${debugger:?}" kube::test::get_object_assert "pod/${debugger:?}" "{{${image_field:?}}}" 'busybox' kube::test::get_object_assert "pod/${debugger:?}" '{{.spec.nodeName}}' '127.0.0.1' kube::test::get_object_assert "pod/${debugger:?}" '{{.spec.hostIPC}}' 'true' kube::test::get_object_assert "pod/${debugger:?}" '{{.spec.hostNetwork}}' 'true' kube::test::get_object_assert "pod/${debugger:?}" '{{.spec.hostPID}}' 'true' kube::test::get_object_assert "pod/${debugger:?}" '{{(index (index .spec.containers 0).volumeMounts 0).mountPath}}' '/host' kube::test::get_object_assert "pod/${debugger:?}" '{{(index .spec.volumes 0).hostPath.path}}' '/' # Clean up # pod.spec.nodeName is set by kubectl debug node which causes the delete to hang, # presumably waiting for a kubelet that's not present. Force the delete. kubectl delete --force pod "${debugger:?}" "${kube_flags[@]:?}" set +o nounset set +o errexit } </ins>

What would you like to be : Add integration tests for Why is this needed: This is part of the kubernetes/enhancements /sig cli /priority important-soon
/assign
/triage accepted


Question: // Update actual state of world to reflect volume is globally mounted markDeviceMappedErr := actualStateOfWorld.MarkDeviceAsMounted( <del> volumeToMount.VolumeName) </del> <ins> volumeToMount.VolumeName, devicePath) </ins> if markDeviceMappedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateErrorDetailed("MapVolume.MarkDeviceAsMounted failed", markDeviceMappedErr)

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: creds := provider.Provide() <del> if len(creds) != len(result) { t.Errorf("Unexpected list: %v, expected length %d", creds, len(result)) </del> <ins> if len(creds) != len(result)+1 { t.Errorf("Unexpected list: %v, expected length %d", creds, len(result)+1) </ins> } for _, cred := range creds { <del> if cred.Username != "foo" { </del> <ins> if cred.Username != "" && cred.Username != "foo" { </ins> t.Errorf("expected 'foo' for username, saw: %v", cred.Username) } <del> if cred.Password != "bar" { </del> <ins> if cred.Password != "" && cred.Password != "bar" { </ins> t.Errorf("expected 'bar' for password, saw: %v", cred.Username) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently we cannot pull image from Azure Container Registry repo which provides anonymous pull access since when do image pull, it will always use service pincipal for .azurecr. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: }, } <del> doTestPlugin(t, volume.NewSpecFromPersistentVolume(vol, false)) </del> <ins> doTestPlugin(t, volume.NewSpecFromPersistentVolume(vol, false), "localhost:/somepath") </ins> } func TestPersistentClaimReadOnlyFlag(t *testing.T) {

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: // API server does this for new containers, but we repeat this logic in Kubelet // for containers running on existing Kubernetes clusters. if cpuRequest.IsZero() && !cpuLimit.IsZero() { <del> cpuShares = milliCPUToShares(cpuLimit.MilliValue()) </del> <ins> cpuShares = int64(cm.MilliCPUToShares(cpuLimit.MilliValue())) </ins> } else { <del> // if cpuRequest.Amount is nil, then milliCPUToShares will return the minimal number </del> <ins> // if cpuRequest.Amount is nil, then MilliCPUToShares will return the minimal number </ins> // of CPU shares. <del> cpuShares = milliCPUToShares(cpuRequest.MilliValue()) </del> <ins> cpuShares = int64(cm.MilliCPUToShares(cpuRequest.MilliValue())) </ins> } resources.CpuShares = cpuShares if memLimit != 0 {

Deployment of a POD onto a machine with 488 cores fails if the POD gets deployed with QoS Guaranteed or Burstable when cpu request is larger than 256. Kubelet Error Trace: RunContainerError: failed to start container "4ad8bf2f988c9c387d8136ac664caf4af80648ac436e92dac6ecd1f9aece9144": Error response from daemon: OCI runtime create failed: starting container process caused " container init caused " setting cgroup config for procHooks process caused "The maximum allowed cpu-shares is """: unknown Normal start of the container with the right number of cpu-shares set yo the cgroup the POD runs in. Install kubernetes on a machine with more than 300 cores and deploy a POD with QoS Guaranteed or Burstable where the cpu-request value is set to more than 256 cpus. Deployment without CPU request and limits works. Deployments with cpu-request <= 256 works as well. I found this change: Unfortunately the changed code does not get used in all cases - is not used in my case. Kubernetes version (use ): Kubernetes v1.19.8 Cloud provider or hardware configuration: AWS / EC2 u- (6 TB memory / 488 vCPUS) OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp1" VARIANT_ID="chost" Kernel (e.g. ): Linux ip-10-250-0- 4.12.14-197.72-default Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Hi thanks for raising the issue. Can you share the pod spec(maybe the part that includes CPU shares) that caused the issue?

Also which container runtime are you using? And its version?
I think might be related to the error message you're seeing. That repo as you can see is archived so I think you should try upgrading your container runtime. Also in my understanding, CPU share is a relative value that tells kernel scheduler how to schedule the cpu time slice to each task. In my understanding we should expect a large cpu quota here, rather than a large CPU share. Just thinking loud.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. Kernel code where max limits are set: /triage accepted /remove-lifecycle stale It seems we are only clamping for stuff under , I don't know that this is being applied to all pods. /cc I think there's two things we should do: this limit more clearly the clamping to ensure we don't run into this error /kind documentation /help Help requested with improving the docs.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. /remove-lifecycle stale Unfortunately the changed code does not get used in all cases - is not used in my case. Update the clamping to ensure we don't run into this error /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the issue is in the Kubelet. The kernel uses the range for , but the kubelet can generate and request values outside of the valid range. When using cgroupfs, the kernel automatically clamps the value.

Instead systemd has an additional check and refuses to use an invalid value (even if the kernel would accept and clamp it). The clamping is just to make sure the systemd driver works in the same way as cgroupfs, but now any value set above the valid range will get assigned the same number of . would you mind opening a PR to add the clamping also for your case?
I would like to take this one by opening a PR for the clamping issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
have you had a chance to look at this issue?
I just created the PR but not sure if I it to the right place. Could you take a look?
/remove-lifecycle stale
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: Attach:          true, SELinuxRelabel:  true, SupportsMetrics: false, <ins> FSGroup:         true, </ins> } }

What happened: For some filesystems recursively applying an SELinux label through chcon or the GID permission is expensive (30 mins in some cases). There is better strategies for the recursive change and the call should be available through FLEX API. What you expected to happen: FLEX has a shell script call to label or set GID for a volume. Notes-- The fix should be backwards compatible.. If the shell call to perform the permission change is absent, then we fall back on the original method.


Question: } defer func() { <del> podClient.Delete(config.Prefix+"-injector", nil) </del> <ins> podClient.Delete(podName, nil) </ins> }() injectPod, err := podClient.Create(injectPod)

Failing Job: These tests started flaking recently (they were consistently green for a long time): Mounted volume expand[Slow] Should verify mounted devices can be resized Volume expand [Slow] Verify if editing PVC allows resize Subpath [Volume type: gcePDPVC] should support restarting containers using file as subpath [Slow] Subpath [Volume type: gcePDPVC] should fail for new directories when readOnly specified in the volumeSource Subpath [Volume type: gcePDPVC] should fail if subpath with backstepping is outside the volume [Slow] Volumes PD should be mountable with ext4 Affected runs: https://k8s- https://k8s- https://k8s- /kind bug /sig storage /priority important-soon /milestone v1.12 /cc - Could you PTAL or delegate? Thanks.
Volume expansion tests have been fixed by
/assign can you investigate the remaining failures?
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently: https://k8s- https://k8s- /reopen
I think the latest failures are caused by gce API rate limiting: /assign
It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration
/sig scalability
/kind failing-test
/status approved-for-milestone Friendly ping. This is affecting release-blocking scalability job.
hey what needs to be done here? by whom?
Could you comment on what needs to be done here and by whom?
can you guys take a look at the API quota issue?

This is happening consistently in the scale test environment
as Code Freeze is today, this needs to be of priority . Please triage! cc
It looks like the latest run on 9/1 has passed.
Marking closed as the fixes have been merged and the test runs are now passing /close
Closing this issue. <detailsIn response to : Marking closed as the fixes have been merged and the test runs are now passing /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: params+=("--agent-service-account=konnectivity-agent") params+=("--kubeconfig=/etc/srv/kubernetes/konnectivity-server/kubeconfig") params+=("--authentication-audience=system:konnectivity-server") <ins> params+=("--kubeconfig-qps=75") params+=("--kubeconfig-burst=150") </ins> konnectivity_args="" for param in "${params[@]}"; do konnectivity_args+=", "${param}""

After the gce-master-scale-performance started failing with errors like: https://k8s- In a sample run I see a lot of errors like: in konnectivity-server logs: Looks like we have to increase QPS limit for konnectivity-server, do we have a flag (kube-env?) for that? /cc /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/priority critical-urgent
We may need to re-open this, if there are still uncovered problems (or if we need to re-tune) after watching next test runs.
Wow, thanks a lot for such a rapid fix. Much appreciated!


Question: return nil } <del> keyCert.CertFile = path.Join(s.ServerCert.CertDirectory, s.ServerCert.PairName+".crt") keyCert.KeyFile = path.Join(s.ServerCert.CertDirectory, s.ServerCert.PairName+".key") canReadCertAndKey, err := certutil.CanReadCertAndKey(keyCert.CertFile, keyCert.KeyFile) if err != nil { return err </del> <ins> canReadCertAndKey := false if len(s.ServerCert.CertDirectory) > 0 { if len(s.ServerCert.PairName) == 0 { return fmt.Errorf("PairName is required if CertDirectory is set") } keyCert.CertFile = path.Join(s.ServerCert.CertDirectory, s.ServerCert.PairName+".crt") keyCert.KeyFile = path.Join(s.ServerCert.CertDirectory, s.ServerCert.PairName+".key") if canRead, err := certutil.CanReadCertAndKey(keyCert.CertFile, keyCert.KeyFile); err != nil { return err } else { canReadCertAndKey = canRead } </ins> } if !canReadCertAndKey { // add either the bind address or localhost to the valid alternates bindIP := s.BindAddress.String()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: export PROJECT="k8s-jkns-e2e-gke-prod" export CLOUDSDK_BUCKET="gs://cloud-sdk-testing/rc" export JENKINS_USE_SERVER_VERSION="y" <ins> export CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER="https://container.googleapis.com/" </ins> export ZONE="asia-east1-b" - 'gke-prod-parallel': description: 'Run E2E tests on GKE prod endpoint in parallel.'

With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good.

FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?


Question: -v "${REPO_DIR}":/go/src/k8s.io/kubernetes  -v "${KUBE_JUNIT_REPORT_DIR}":/workspace/artifacts  --env REPO_DIR="${REPO_DIR}"  <del> -i gcr.io/google_containers/kubekins-test:0.5  </del> <ins> -i gcr.io/google_containers/kubekins-test:0.6  </ins> bash -c "cd kubernetes && ./hack/jenkins/test-dockerized.sh"

It seems like an issue similar to the problem with API server. I'll extend the timeout in the similar fashion. cc
Extending timeout didn't help. It's probably something else.
cc
I think the logs make it pretty clear what went wrong: or does one of you have time to fix this?
The Jenkins docker test image probably needs netcat to be installed. Should be an easy fix; I can take this.
Thanks!
Ugh... neverending story.
This seems like a different problem.
Oh, I see, this actually is the same, but harder: I think the most obvious thing to do is to let kubectl select the port and report back?
reports more occurances. Netcat somehow is no longer installed?
those two are older builds, using , which didn't have netcat yet.
ah thanks!
are you still working on this? Would you like me to take it?
I'm not right now, sorry, feel free to take it.


Question: delete(pk.podTerminationMap, uid) } <del> // checkAndMarkPodPendingTerminationByPod checks to see if the pod is being // killed and returns true if it is, otherwise the pod is added to the map and // returns false func (pk *podKillerWithChannel) checkAndMarkPodPendingTerminationByPod(podPair *kubecontainer.PodPair) bool { </del> <ins> // KillPod sends pod killing request to the killer after marks the pod // unless the given pod has been marked to be killed func (pk *podKillerWithChannel) KillPod(podPair *kubecontainer.PodPair) { </ins> pk.podKillingLock.Lock() defer pk.podKillingLock.Unlock() var apiPodExists bool

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: } return rows, nil } <ins> func printResourceQuota(resourceQuota *api.ResourceQuota, options printers.PrintOptions) ([]metav1beta1.TableRow, error) { row := metav1beta1.TableRow{ Object: runtime.RawExtension{Object: resourceQuota}, } resources := make([]api.ResourceName, 0, len(resourceQuota.Status.Hard)) for resource := range resourceQuota.Status.Hard { resources = append(resources, resource) } sort.Sort(SortableResourceNames(resources)) requestColumn := bytes.NewBuffer([]byte{}) limitColumn := bytes.NewBuffer([]byte{}) for i := range resources { w := requestColumn resource := resources[i] usedQuantity := resourceQuota.Status.Used[resource] hardQuantity := resourceQuota.Status.Hard[resource] // use limitColumn writer if a resource name prefixed with "limits" is found if pieces := strings.Split(resource.String(), "."); len(pieces) > 1 && pieces[0] == "limits" { w = limitColumn } fmt.Fprintf(w, "%s: %s/%s, ", resource, usedQuantity.String(), hardQuantity.String()) } age := translateTimestampSince(resourceQuota.CreationTimestamp) row.Cells = append(row.Cells, resourceQuota.Name, age, strings.TrimSuffix(requestColumn.String(), ", "), strings.TrimSuffix(limitColumn.String(), ", ")) return []metav1beta1.TableRow{row}, nil } func printResourceQuotaList(list *api.ResourceQuotaList, options printers.PrintOptions) ([]metav1beta1.TableRow, error) { rows := make([]metav1beta1.TableRow, 0, len(list.Items)) for i := range list.Items { r, err := printResourceQuota(&list.Items[i], options) if err != nil { return nil, err } rows = append(rows, r...) } return rows, nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature isn't very useful as it only shows the name and age of the resourcequota resource. It'd be nice if it also shows the current/allocated usage for each, like this: Each resourcequota summary is of the format: . I think it would be a useful feature. I wouldn't mind taking a crack at implementing it if it's deemed worth by the maintainers. Thanks.
/sig cli
/assign code-sleuth
GitHub didn't allow me to assign the following users: code-sleuth. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign code-sleuth Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign juanvallejo


Question: nodeSnapshot.generation = cache.headNode.info.Generation } <del> if len(nodeSnapshot.nodeInfoMap) > len(cache.nodes) { </del> <ins> // Comparing to pods in nodeTree. // Deleted nodes get removed from the tree, but they might remain in the nodes map // if they still have non-deleted Pods. if len(nodeSnapshot.nodeInfoMap) > cache.nodeTree.numNodes { </ins> cache.removeDeletedNodesFromSnapshot(nodeSnapshot) updateAllLists = true }

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: requestGB := int(volume.RoundUpSize(requestBytes, 1024*1024*1024)) volumeOptions := &aws.VolumeOptions{ CapacityGB: requestGB, <del> Tags:       &tags, </del> <ins> Tags:       tags, </ins> } name, err := cloud.CreateDisk(volumeOptions)

We should tag EBS volumes created by k8s with our tags, just like we (should) tag everything else we create/own.
Is there any documentation yet on automatic EBS volume creation?
from a user perspective, they are not AWS specific. That said, I don't know that we have clear user-facing docs. Here's an example though, from my (very) WIP Ubernetes-Lite docs: ... The PV is also tagged with the zone & region is is in (us-central1-a): Create a pod that uses the PVC:
It doesn't seem to cleanup the volume when the PV is deleted... probably intended behavior. Any way to tell it to automatically delete the volume on PV delete?
- I'm not sure about foo - hopefully that'll be resolved with docs. I think the failure to delete the volume is actually pretty important. Do you want to open an issue? It's not AWS specific (and this particular issue was auto-closed when the PR merged)
yeah I will open an issue later today. Issue seems to still be happening - the PV is never deleted. On Sat, Mar 12, 2016, 6:24 AM Justin Santa Barbara wrote:


Question: if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } <del> if filterPlugin.numFilterCalled != tt.expectFilterNumCalled { t.Errorf("Expected the filter plugin to be called %v times, but got %v.", tt.expectFilterNumCalled, filterPlugin.numFilterCalled) </del> <ins> if numFilterCalled := atomic.LoadInt32(&filterPlugin.numFilterCalled); numFilterCalled != tt.expectFilterNumCalled { t.Errorf("Expected the filter plugin to be called %v times, but got %v.", tt.expectFilterNumCalled, numFilterCalled) </ins> } if numScoreCalled := atomic.LoadInt32(&scorePlugin.numScoreCalled); numScoreCalled != tt.expectScoreNumCalled { t.Errorf("Expected the score plugin to be called %v times, but got %v.", tt.expectScoreNumCalled, numScoreCalled)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: integration-master Which test(s) are flaking: TestPostFilterPlugin Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /remove-sig testing
/assign