Question:
namespaceSelector

NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is other cluster scoped resource, it is not subjected to the webhook.

NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.

 For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {
 "matchExpressions": [

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: Namespace: namespace.Name, Prefix:    "configmap", } <del> defer volume.TestCleanup(f, config) </del> configMap := &v1.ConfigMap{ TypeMeta: metav1.TypeMeta{ Kind:       "ConfigMap",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: <ins> reviewers: - lavalamp - smarterclayton - thockin approvers: - lavalamp - smarterclayton - thockin </ins>

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: maxBudget time.Duration } <del> func newTimeBudget(stopCh <-chan struct{}) *timeBudget { result := &timeBudget{ </del> <ins> func newTimeBudget(stopCh <-chan struct{}) timeBudget { result := &timeBudgetImpl{ </ins> budget:    time.Duration(0), refresh:   refreshPerSecond, maxBudget: maxBudget,

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: // NewBase64Plugin is a constructor for Base64Plugin. func NewBase64Plugin(t testing.TB, socketPath string) *Base64Plugin { <del> server := grpc.NewServer() </del> <ins> server := grpc.NewServer( grpc.UnaryInterceptor( func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) { if val := metadata.ValueFromIncomingContext(ctx, ":authority"); len(val) != 1 || val[0] != "localhost" { t.Errorf("wanted localhost authority, got: %v", val) } return handler(ctx, req) }, ), ) </ins> result := &Base64Plugin{ grpcServer: server, mu:         &sync.Mutex{},

KMS Plugins require communication via grpc UDS. Currently, the k8s grpc client sets the authority header to the socket path which is marked as invalid preventing successful socket communication. I expect a custom KMS provider plugin to be sent valid authority headers so that successful communication can be established between the k8s and grpc client/server. Create a grpc server according to the and attempt to establish socket communication. This issue has been resolved in for device plugins and CSI. The same solution will likely fix this issue as well. I will create a PR for this and link it here after. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig auth /triage accepted


Question: <del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates import ( "bytes" "crypto/x509" "fmt" "io/ioutil" "sync/atomic" "time" "k8s.io/client-go/util/cert" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" "k8s.io/klog" ) // FileRefreshDuration is exposed so that integration tests can crank up the reload speed. var FileRefreshDuration = 1 * time.Minute // Listener is an interface to use to notify interested parties of a change. type Listener interface { // Enqueue should be called when an input may have changed Enqueue() } // Notifier is a way to add listeners type Notifier interface { // AddListener is adds a listener to be notified of potential input changes AddListener(listener Listener) } // ControllerRunner is a generic interface for starting a controller type ControllerRunner interface { // RunOnce runs the sync loop a single time.  This useful for synchronous priming RunOnce() error // Run should be called a go .Run Run(workers int, stopCh <-chan struct{}) } // DynamicFileCAContent provies a CAContentProvider that can dynamically react to new file content // It also fulfills the authenticator interface to provide verifyoptions type DynamicFileCAContent struct { name string // filename is the name the file to read. filename string // caBundle is a caBundleAndVerifier that contains the last read, non-zero length content of the file caBundle atomic.Value listeners []Listener // queue only ever has one item, but it has nice error handling backoff/retry semantics queue workqueue.RateLimitingInterface } var _ Notifier = &DynamicFileCAContent{} var _ CAContentProvider = &DynamicFileCAContent{} var _ ControllerRunner = &DynamicFileCAContent{} type caBundleAndVerifier struct { caBundle      []byte verifyOptions x509.VerifyOptions } // NewDynamicCAContentFromFile returns a CAContentProvider based on a filename that automatically reloads content func NewDynamicCAContentFromFile(purpose, filename string) (*DynamicFileCAContent, error) { if len(filename) == 0 { return nil, fmt.Errorf("missing filename for ca bundle") } name := fmt.Sprintf("%s::%s", purpose, filename) ret := &DynamicFileCAContent{ name:     name, filename: filename, queue:    workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), fmt.Sprintf("DynamicCABundle-%s", purpose)), } if err := ret.loadCABundle(); err != nil { return nil, err } return ret, nil } // AddListener adds a listener to be notified when the CA content changes. func (c *DynamicFileCAContent) AddListener(listener Listener) { c.listeners = append(c.listeners, listener) } // loadCABundle determines the next set of content for the file. func (c *DynamicFileCAContent) loadCABundle() error { caBundle, err := ioutil.ReadFile(c.filename) if err != nil { return err } if len(caBundle) == 0 { return fmt.Errorf("missing content for CA bundle %q", c.Name()) } // check to see if we have a change. If the values are the same, do nothing. if !c.hasCAChanged(caBundle) { return nil } caBundleAndVerifier, err := newCABundleAndVerifier(c.Name(), caBundle) if err != nil { return err } c.caBundle.Store(caBundleAndVerifier) for _, listener := range c.listeners { listener.Enqueue() } return nil } // hasCAChanged returns true if the caBundle is different than the current. func (c *DynamicFileCAContent) hasCAChanged(caBundle []byte) bool { uncastExisting := c.caBundle.Load() if uncastExisting == nil { return true } // check to see if we have a change. If the values are the same, do nothing. existing, ok := uncastExisting.(*caBundleAndVerifier) if !ok { return true } if !bytes.Equal(existing.caBundle, caBundle) { return true } return false } // RunOnce runs a single sync loop func (c *DynamicFileCAContent) RunOnce() error { return c.loadCABundle() } // Run starts the kube-apiserver and blocks until stopCh is closed. func (c *DynamicFileCAContent) Run(workers int, stopCh <-chan struct{}) { defer utilruntime.HandleCrash() defer c.queue.ShutDown() klog.Infof("Starting %s", c.name) defer klog.Infof("Shutting down %s", c.name) // doesn't matter what workers say, only start one. go wait.Until(c.runWorker, time.Second, stopCh) // start timer that rechecks every minute, just in case.  this also serves to prime the controller quickly. _ = wait.PollImmediateUntil(FileRefreshDuration, func() (bool, error) { c.queue.Add(workItemKey) return false, nil }, stopCh) // TODO this can be wired to an fsnotifier as well. <-stopCh } func (c *DynamicFileCAContent) runWorker() { for c.processNextWorkItem() { } } func (c *DynamicFileCAContent) processNextWorkItem() bool { dsKey, quit := c.queue.Get() if quit { return false } defer c.queue.Done(dsKey) err := c.loadCABundle() if err == nil { c.queue.Forget(dsKey) return true } utilruntime.HandleError(fmt.Errorf("%v failed with : %v", dsKey, err)) c.queue.AddRateLimited(dsKey) return true } // Name is just an identifier func (c *DynamicFileCAContent) Name() string { return c.name } // CurrentCABundleContent provides ca bundle byte content func (c *DynamicFileCAContent) CurrentCABundleContent() (cabundle []byte) { return c.caBundle.Load().(*caBundleAndVerifier).caBundle } // VerifyOptions provides verifyoptions compatible with authenticators func (c *DynamicFileCAContent) VerifyOptions() x509.VerifyOptions { return c.caBundle.Load().(*caBundleAndVerifier).verifyOptions } // newVerifyOptions creates a new verification func from a file.  It reads the content and then fails. // It will return a nil function if you pass an empty CA file. func newCABundleAndVerifier(name string, caBundle []byte) (*caBundleAndVerifier, error) { if len(caBundle) == 0 { return nil, fmt.Errorf("missing content for CA bundle %q", name) } // Wrap with an x509 verifier var err error verifyOptions := defaultVerifyOptions() verifyOptions.Roots, err = cert.NewPoolFromBytes(caBundle) if err != nil { return nil, fmt.Errorf("error loading CA bundle for %q: %v", name, err) } return &caBundleAndVerifier{ caBundle:      caBundle, verifyOptions: verifyOptions, }, nil } // defaultVerifyOptions returns VerifyOptions that use the system root certificates, current time, // and requires certificates to be valid for client auth (x509.ExtKeyUsageClientAuth) func defaultVerifyOptions() x509.VerifyOptions { return x509.VerifyOptions{ KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, } } </del>

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: SelectorParam("kubernetes.io/cluster-service=true"). ResourceTypeOrNameArgs(false, []string{"services"}...). Latest() <del> b.Do().Visit(func(r *resource.Info, err error) error { </del> <ins> err = b.Do().Visit(func(r *resource.Info, err error) error { </ins> if err != nil { return err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: reports that the cluster is running and returns exit code 0. correctly identifies that the cluster is not running and exists with exit code 1. What you expected to happen: should fail when the cluster is not running. How to reproduce it (as minimally and precisely as possible): Start and stop a kubernetes cluster using minikube and then type kubectl cluster-info. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-06T20:53:14Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: Local OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Ubuntu16-HP-800 4.10.0-35-generic ~16.04.1-Ubuntu SMP Wed Sep 13 09:02:42 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: snap for kubectl. minikube local for k8s. Others:
/assign /sig cli


Question: curl -sS -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -M -N # Install salt-api <del> # </del> <ins> # # This is used to provide the network transport for salt-api yum install -y python-cherrypy </ins> # This is used to inform the cloud provider used in the vagrant cluster yum install -y salt-api # Set log level to a level higher than "info" to prevent the message about

I use ubuntu 13.04 system. (kernel version 3.8.0-44-generic, vagrant version 1.6.5, virtualbox version : 4.3.20) I download the kubernetes tar file version 0.5.4. and then, I run script file "/cluster/kube-" master, minion-1,2,3 each machine instance has been created. but when "" script file run, it could loop forever. the message is "Waiting for each minion to be registered with cloud provider" maybe during provision of cluster, it occurs some problems. but I don't know what is problem. please let me know to solve problem.
Related to . it loops trying to detect minions, but master node can't find any. I tried with the git master branch, various commits picked from the time doc has been updated, v0.6.0, v0.5.5, v0.5.4, all without success. I find it hard to believe that it's broken since that time, but I can't figure out what else could cause that problem.
I'm currently tracking this down as well. I believe it is related to commit (which was cherry-picked from ). If you run do you see that the kubelet is running, but is unable to send events to the kube-apiserver? Probably due to x509 certificate signed by an unknown authority? Also, please confirm that no minions at all register by running could you try running with v0.5.0? I'm not at my desk at the moment to work on a proper fix just yet, assuming the commit I mentioned is the problem.
yes, I ran list minions everytime, and the list was empty. Tried to build v0.5, but got the error . Getting late here, will try again another time. Thanks for looking into it!
I've made some headway here today. There are a few different things at play, I think. 1) The kube-apiserver is provided /dev/null as tokenauthfile. It should just be omitted. 2) The kubelet only has an empty /var/lib/kubelet/kubernetes_auth file. It should be populated appropriately with the CA and certs 3) 's on the group pointed me to the last piece of the puzzle.

It appears that the minions are registered with salt. For some reason on the master, the python cherrypy module is missing, and as far as I can tell, the salt rest interface never comes up during the provisioning. I, for now, hacked in a to the script and that appeared to work. I'll try to come up with a full solution and submit a PR.
Seconded. I independently went through the same steps as ryaneleary. I don't have too much to add on top of his comments. There are some bits of authorization around the apiserver, but they seem unrelated to minion registration. Nonetheless, I did try to implement token generation for it for Vagrant as part of my flailing around with this problem. I might send a PR for that in case it's useful on its own. The real issue, though, was the salt-api service. As ryan mentioned, that service depends on python-cherrypy, which somehow wasn't installed as part of "yum install -y salt-api" in the setup scripts. I tried to track down what happened there. All of the RPM spec files I can find for the salt-api package are (a) ancient and (b) contain a "Requires: python-cherrypy" line. However, the package that's actually installed on my system is versioned 2014.7.0-3.fc20 and does not have the dependency. I can't find the corresponding SRPM for it. yumdownloader --source gives me an older version. So, in summary, this looks to me like a Fedora bug. I'll file a bug with them once I figure out how to access their bug tracker. I think it makes sense to apply ryan's solution in the meantime, so that no one else trips over this issue.
Upstream bug report:
Shouldn't we move to CoreOS for the vagrant system? I do not understand why Fedora, which is not supported anymore, was chosen for this example :)
/cc
I can try to reproduce and fix-up any issue described here on Monday. The environment was stable prior to the holiday when I was working from HEAD, so it should be quick to nail down. - can you elaborate on your Fedora not being supported comment? The Fedora based environment has been supported for months now.

I have been debating about adding CentOS support, but when I originally authored the environment, it was intended to help support compatibility in a distribution in the broader RHEL ecosystem. If someone wants to submit a PR to support an additional Vagrant environment using CoreOS that is fine, but we need someone to take ownership of supporting it. We at Red Hat will probably add an Atomic host (CentOS 7 Atomic) at some point in the future as a supported Vagrant provider. The Vagrantfile structure is setup today to allow you to provide other boxes, but it would also require custom provisioners, etc.
Sorry, please discard my comment, got it wrong.
There appears to have been a packaging change in that caused the module to not get included, and as a result, the cloud_provider used in the vagrant env is not able to contact the salt-server to find out the available minions. Submitting a PR now. Thanks for reporting the issue.


Question: {ObjectMeta: metav1.ObjectMeta{Name: "a", UID: types.UID("a"), Labels: map[string]string{"app": "foo"}}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}}, {ObjectMeta: metav1.ObjectMeta{Name: "b", UID: types.UID("b"), Labels: map[string]string{"app": "foo"}}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}}}, pdbs: []*policy.PodDisruptionBudget{ <del> {Spec: policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "foo"}}}, Status: policy.PodDisruptionBudgetStatus{DisruptionsAllowed: 1, DisruptedPods: map[string]metav1.Time{"a": {Time: time.Now()}}}}}, </del> <ins> {Spec: policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "foo"}}}, Status: policy.PodDisruptionBudgetStatus{DisruptionsAllowed: 1, DisruptedPods: map[string]metav1.Time{"b": {Time: time.Now()}}}}}, </ins> expected:                map[string]victims{"machine1": {pods: sets.NewString("a", "b"), numPDBViolations: 0}}, expectedNumFilterCalled: 3, }, <ins> { name: "preemption with violation of the pdb with pod whose eviction was processed, the victim which belongs to DisruptedPods is treated as 'nonViolating'", registerPlugins: []st.RegisterPluginFunc{ st.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), st.RegisterPluginAsExtensions(noderesources.FitName, noderesources.NewFit, "Filter", "PreFilter"), st.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), }, nodes: []string{"machine1"}, pod:   &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "pod1", UID: types.UID("pod1")}, Spec: v1.PodSpec{Containers: veryLargeContainers, Priority: &highPriority}}, pods: []*v1.Pod{ {ObjectMeta: metav1.ObjectMeta{Name: "a", UID: types.UID("a"), Labels: map[string]string{"app": "foo"}}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}}, {ObjectMeta: metav1.ObjectMeta{Name: "b", UID: types.UID("b"), Labels: map[string]string{"app": "foo"}}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}}, {ObjectMeta: metav1.ObjectMeta{Name: "c", UID: types.UID("c"), Labels: map[string]string{"app": "foo"}}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}}}, pdbs: []*policy.PodDisruptionBudget{ {Spec: policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"app": "foo"}}}, Status: policy.PodDisruptionBudgetStatus{DisruptionsAllowed: 1, DisruptedPods: map[string]metav1.Time{"c": {Time: time.Now()}}}}}, expected:                map[string]victims{"machine1": {pods: sets.NewString("a", "b", "c"), numPDBViolations: 1}}, expectedNumFilterCalled: 4, }, </ins> } labelKeys := []string{"hostname", "zone", "region"} for _, test := range tests {

In one recent fix (), we enhanced the preemption logic to respect . But there is a glitch: if we changed the DisruptedPods in L1499 from "a" to "b", the test would fail it would also fail if we change the pods (i.e., []Pod{"b", "a"}) ordering in L1496~L1497 This is a problem we need to fix. On the other hand, I'm thinking about the real semantics of . According to the docs, if a pod is matched by the "DisruptedPods" list, it denotes the pod is about to be evicted by API server as it at least has violated one disruption budget. Given this fact, IMO in scheduler preemption, we should treat this kind of pod as "nonViolating" in all cases - no matter the current is negative or not. So that the scheduler would treat them as preemption candidates. /sig scheduling
/assign
cc/


Question: gem 'fluentd', '<=1.2.4' gem 'activesupport', '~>5.2.1' <del> gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0' gem 'fluent-plugin-elasticsearch', '~>2.11.5' gem 'fluent-plugin-systemd', '~>1.0.1' </del> <ins> gem 'fluent-plugin-concat', '~>2.3.0' </ins> gem 'fluent-plugin-detect-exceptions', '~>0.0.11' <del> gem 'fluent-plugin-prometheus', '~>1.0.1' </del> <ins> gem 'fluent-plugin-elasticsearch', '~>2.11.5' gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0' </ins> gem 'fluent-plugin-multi-format-parser', '~>1.0.0' <ins> gem 'fluent-plugin-prometheus', '~>1.0.1' gem 'fluent-plugin-systemd', '~>1.0.1' </ins> gem 'oj', '~>3.6.5'

Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective.

I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue.

If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together?

The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.


Question: import ( "errors" "fmt" <ins> "sort" </ins> "sync" "github.com/coreos/go-etcd/etcd"

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package unstructured import ( "testing" "github.com/stretchr/testify/assert" ) func TestNilUnstructuredContent(t *testing.T) { var u Unstructured uCopy := u.DeepCopy() content := u.UnstructuredContent() expContent := make(map[string]interface{}) assert.EqualValues(t, expContent, content) assert.Equal(t, uCopy, &u) } </ins>

What happened: In I introduced an optimization - do a deep copy instead of to and from JSON roundtrip to convert anything that implements . I then discovered that the method that is used there in both and may mutate the original object. This is problematic because previously (before ) there was no mutation and because this is unexpected and may lead to data races - it is bad behaviour to mutate original object when you just want a copy of it. This issue was fixed by . However the underlying issue is not fixed here - is brittle and dangerous. Method name does not imply that it mutates data when you call it. And godoc does not mention that either: Something needs to be done about it IMO. This method is misused in these places (mutation side effect is not intended): Also implementation in does not implement the behaviour required by godoc in . What you expected to happen: There should be a method that returns the current unstructured contents without mutation and perhaps not providing the write-through effect. /kind bug /sig api-machinery
/cc
Is this still required and could I work on it? I'm looking for something to work on as a new contributor :)
Yes and yes! :)
Perfect, I'll get started on it!
I've got out. I'd appreciate a review :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: pv.Spec.AccessModes = r.plugin.GetAccessModes() } <del> return pv, os.MkdirAll(pv.Spec.HostPath.Path, 0750) </del> <ins> if err := os.MkdirAll(pv.Spec.HostPath.Path, 0750); err != nil { return nil, err } if selinux.GetEnabled() { err := selinux.SetFileLabel(pv.Spec.HostPath.Path, config.KubeletContainersSharedSELinuxLabel) if err != nil { return nil, fmt.Errorf("failed to set selinux label for %q: %v", pv.Spec.HostPath.Path, err) } } return pv, nil </ins> } // hostPathDeleter deletes a hostPath PV from the cluster.

crio-e2e tests The following tests fail when run on a node with SELinux enabled: Probably since these tests were . No response The host path volumes created under /tmp/hostpathpv don't have the right SELinux label set for access by containers started during the test. No response_ /sig node
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodListQuantity(f, "type=Testing", 0)) framework.ExpectNoError(err, "found a pod(s)") }) <ins> ginkgo.It("should run through the lifecycle of Pods and PodStatus", func() { podResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"} testNs := f.Namespace.Name testPodName := "pod-test" testPodImage := "nginx" testPodImage2 := "httpd" testPodLabels := map[string]string{"test-pod-static": "true"} testPodLabelsFlat := "test-pod-static=true" testPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   testPodName, Labels: testPodLabels, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  testPodName, Image: testPodImage, }, }, }, } ginkgo.By("creating a Pod with a static label") _, err := f.ClientSet.CoreV1().Pods(testNs).Create(context.TODO(), &testPod, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Pod %s in namespace %s", testPodName, testNs) ginkgo.By("setting up a watch for the Pod") podWatchTimeoutSeconds := int64(180) podWatch, err := f.ClientSet.CoreV1().Pods(testNs).Watch(context.TODO(), metav1.ListOptions{LabelSelector: testPodLabelsFlat, TimeoutSeconds: &podWatchTimeoutSeconds}) framework.ExpectNoError(err, "failed to set up watch for Pod %s in namespace %s", testPodName, testNs) podWatchChan := podWatch.ResultChan() ginkgo.By("watching for Pod to be ready") eventFound := false for watchEvent := range podWatchChan { if podWatchEvent, ok := watchEvent.Object.(*v1.Pod); ok && podWatchEvent.Status.Phase == v1.PodRunning { eventFound = true break } } framework.ExpectEqual(eventFound, true, "failed to find running Pod %s in namespace %s", testPodName, testNs) ginkgo.By("patching the Pod with a new Label and updated data") podPatch, err := json.Marshal(v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"test-pod": "patched"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{{ Name:  testPodName, Image: testPodImage2, }}, }, }) framework.ExpectNoError(err, "failed to marshal JSON patch for Pod") _, err = f.ClientSet.CoreV1().Pods(testNs).Patch(context.TODO(), testPodName, types.StrategicMergePatchType, []byte(podPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch Pod %s in namespace %s", testPodName, testNs) ginkgo.By("getting the Pod and ensuring that it's patched") pod, err := f.ClientSet.CoreV1().Pods(testNs).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch Pod %s in namespace %s", testPodName, testNs) framework.ExpectEqual(pod.ObjectMeta.Labels["test-pod"], "patched", "failed to patch Pod - missing label") framework.ExpectEqual(pod.Spec.Containers[0].Image, testPodImage2, "failed to patch Pod - wrong image") ginkgo.By("getting the PodStatus") podStatusUnstructured, err := dc.Resource(podResource).Namespace(testNs).Get(context.TODO(), testPodName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "failed to fetch PodStatus of Pod %s in namespace %s", testPodName, testNs) podStatusBytes, err := json.Marshal(podStatusUnstructured) framework.ExpectNoError(err, "failed to marshal unstructured response") var podStatus v1.Pod err = json.Unmarshal(podStatusBytes, &podStatus) framework.ExpectNoError(err, "failed to unmarshal JSON bytes to a Pod object type") ginkgo.By("replacing the Pod's status Ready condition to False") podStatusUpdated := podStatus podStatusFieldPatchCount := 0 podStatusFieldPatchCountTotal := 2 for pos, cond := range podStatusUpdated.Status.Conditions { if (cond.Type == v1.PodReady && cond.Status == v1.ConditionTrue) || (cond.Type == v1.ContainersReady && cond.Status == v1.ConditionTrue) { podStatusUpdated.Status.Conditions[pos].Status = v1.ConditionFalse podStatusFieldPatchCount++ } } framework.ExpectEqual(podStatusFieldPatchCount, podStatusFieldPatchCountTotal, "failed to patch all relevant Pod conditions") podStatusUpdate, err := f.ClientSet.CoreV1().Pods(testNs).UpdateStatus(context.TODO(), &podStatusUpdated, metav1.UpdateOptions{}) framework.ExpectNoError(err, "failed to update PodStatus of Pod %s in namespace %s", testPodName, testNs) ginkgo.By("check the Pod again to ensure its Ready conditions are False") podStatusFieldPatchCount = 0 podStatusFieldPatchCountTotal = 2 for _, cond := range podStatusUpdate.Status.Conditions { if (cond.Type == v1.PodReady && cond.Status == v1.ConditionFalse) || (cond.Type == v1.ContainersReady && cond.Status == v1.ConditionFalse) { podStatusFieldPatchCount++ } } framework.ExpectEqual(podStatusFieldPatchCount, podStatusFieldPatchCountTotal, "failed to update PodStatus - field patch count doesn't match the total") ginkgo.By("deleting the Pod via a Collection with a LabelSelector") err = f.ClientSet.CoreV1().Pods(testNs).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: testPodLabelsFlat}) framework.ExpectNoError(err, "failed to delete Pod by collection") ginkgo.By("watching for the Pod to be deleted") eventFound = false for watchEvent := range podWatchChan { if watchEvent.Type == watch.Deleted { eventFound = true break } } framework.ExpectEqual(eventFound, true, "failed to find Pod %v event", watch.Deleted) }) </ins> }) func checkPodListQuantity(f *framework.Framework, label string, quantity int) func() (bool, error) {

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: BASH_TARGETS=" update-generated-protobuf update-codegen <ins> update-generated-runtime update-generated-device-plugin </ins> update-generated-docs update-generated-swagger-docs update-swagger-spec

Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.


Question: t.Fatal("New DS has the same UID as the old one!") } <del> err = wait.PollImmediate(100*time.Millisecond, informerSyncTimeout, func() (bool, error) { klog.V(8).Infof("Waiting for queue to have 1 item, currently has: %d", dsc.queue.Len()) return dsc.queue.Len() == 1, nil }) if err != nil { t.Fatalf("Re-creating DS didn't result in new item in the queue: %v", err) } </del> <ins> waitForQueueLength(1, "recreated DS") </ins> ok = dsc.processNextWorkItem() if !ok { t.Fatal("Queue is shutting down!")

Which jobs are failing: TestExpectationsOnRecreate unit test, in along with expectations bugfix during 1.19 and cherry-picked to 1.18/1.17 Which test(s) are failing: Reason for failure: Anything else we need to know: These failures are masked in CI because bazel test runs with flake retries (), so the test can fail 2/3 times and the job can still pass (cc
adding to the milestone for triage, in case this is a real bug and not just a test issue /milestone v1.19
cc
/assign
looks like the panic is due to the test continuing in the !exists case (rather than exiting with a Fatal): however, that still means the test is failing in three distinct places:
it looks like the same failure existed in 1.18, so this probably isn't release blocking, but it does need to be resolved in order for us to stop ignoring unit test flakes in CI runs
it actually looks like this test was during the 1.19 dev cycle and cherry-picked back to 1.18/1.17, which is why release-1.18 is failing :-/
figured it out... the controller sync is updating the DS, which is triggering an update event and adding an extra enqueue which the test did not account for good news is that it appears to be a test-only issue
possible test deflake in


Question: cadvisorapiv1 "github.com/google/cadvisor/info/v1" "k8s.io/klog" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: Request Headers: {% for i, key in ipairs(keys) do %} <del> {{key}}={{headers[key]}} </del> <ins> {% local val = headers[key] %} {% if type(val) == "table" then %} {% for i = 1,#val do %} {{key}}={{val[i]}} {% end %} {% else %} {{key}}={{val}} {% end %} </ins> {% end %} Request Body:

/kind bug Hello, actually this is a bug about not about k8s. I use to observe request headers and found if the request have multiple same headers echoserver will give a wrong result: result: Expected result:
/sig testing
any chance you know some lua?
I never used lua before, but I'm willing to try to fix this issue.


Question: # keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1:10053 >/dev/null - -port=8080

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: continue } resp.Body.Close() <ins> if e, a := item.namespace, simpleStorage.requestedResourceNamespace; e != a { t.Errorf("%v: expected %v, got %v", item.rawQuery, e, a) } </ins> if e, a := item.resourceVersion, simpleStorage.requestedResourceVersion; e != a { t.Errorf("%v: expected %v, got %v", item.rawQuery, e, a) }

is always using the cross namespace context. This needs to be fixed to support scoped watches.


Question: proxyStream(response.ResponseWriter, request.Request, url) } <ins> // trimURLPath trims a URL path. // For paths in the format of "/metrics/xxx", "metrics/xxx" is returned; // For all other paths, the first part of the path is returned. func trimURLPath(path string) string { parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3) if len(parts) == 0 { return path } if parts[0] == "metrics" && len(parts) > 1 { return fmt.Sprintf("%s/%s", parts[0], parts[1]) } return parts[0] } // isLongRunningRequest determines whether the request is long-running or not. func isLongRunningRequest(path string) bool { longRunningRequestPaths := []string{"exec", "attach", "portforward", "debug"} for _, p := range longRunningRequestPaths { if p == path { return true } } return false } </ins> // ServeHTTP responds to HTTP requests on the Kubelet. func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) { defer httplog.NewLogged(req, &w).StacktraceWhen(

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: <del> id: blog-service </del> <ins> apiVersion: v1beta3 </ins> kind: Service <del> apiVersion: v1beta1 port: 80 containerPort: 80 selector: name: blog createExternalLoadBalancer: true </del> <ins> metadata: name: blog-service spec: createExternalLoadBalancer: true ports: - port: 80 selector: name: blog </ins>

This causes . In contrib/git-sync demo, the container voluntarily exits and gets recreated to "periodically" sync the git repo, but the pod is then reported as not ready, The end result is that the service targets at this pod does not get endpoints and thus not working. Thanks to for helping with the debugging.
The container was not running when kubelet reported the status, hence it was considered not ready. Because the containers in a pod share the same fate, the pod was not perceived as not ready. Having short-lived containers as a service endpoint may encounter problems like this. I am not really convinced that this is kubelet's problem. It sounds like should be a cron job...
Forgot to cc
Is it alright if I modify git-sync to make it run a for-loop?
sure, but it would still be nice to able to have a way to have background containers restarting periodically in a pod w/o failing the service.
, will it be useful if we let the user to config which containers will be considered by kubelet when it reports the readiness of the pod?
I am not sure kubelet restart is the right mechanism to run periodic tasks, and in this case, recreating the container isn't particularly efficient. FYI, there is a discussion for adding a controller for cron jobs in .
Thanks Let's forget about letting kubelet restart periodic tasks. In general, if some containers are not critical to a service, ignoring the status of these containers and keeping the service alive will increase the availability of the service.
we'd like to have sidecar containers that aren't critical to a pod's overall functioning. i.e. a sidecar which can crash or become unavailable without pulling the pod from the load balancer. In our case it is a sidecar which delivers some configuration to disk from a notoriously flakey service. We'd like the pod to stay up, the conf it needs is on disk. Should I file a separate issue for this?


Question: sslPorts := getPortSets(annotations[ServiceAnnotationLoadBalancerSSLPorts]) for _, port := range apiService.Spec.Ports { <del> if port.Protocol != v1.ProtocolTCP { return nil, fmt.Errorf("Only TCP LoadBalancer is supported for AWS ELB") </del> <ins> if err := checkProtocol(port, annotations); err != nil { return nil, err </ins> } if port.NodePort == 0 { klog.Errorf("Ignoring port without NodePort defined: %v", port) continue

What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call.

If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!

C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising.

If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .


Question: testutils.SyncInformerFactory(testCtx) defer testutils.CleanupTest(t, testCtx) <ins> defer testCtx.Scheduler.SchedulingQueue.Close() </ins> cs, ns, ctx := testCtx.ClientSet, testCtx.NS.Name, testCtx.Ctx // Create one Node with a taint.

The integration framework runs an apiserver and sometimes other components before each test execution, however, the shutdown is not clean and is leaving a lot of goroutines orphan that keep accumulation between different runs. Some orphan goroutines are connections to etcd, and since the etcd process is shared for all test, we don't know if this can cause contention and flakiness on the job. We are logging the number of goroutines leaked per test binary execution, it is important if we can reduce them and more important we don't regress and this number keeps growing. This issue will record the state as today for reference:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
hmm, not doing better
I'll take the scheduler part for a deep investigate.
For people that want to get access to the stack traces, just apply this patch, so the framework will dump the output of the goroutines after each execution: [x] Apiserver was shutting down the listener before the hooks could finish, hence the controllers couldn't shutdown gracefully since there was no http server [ ] The storage cacher doesn't differentiate between not read and stopping - Missing integration test reproducing the issue, only reproduced with unit tests [ ] admission plugins can't shutdown gracefully - - there is a PR addressing However, most of the contributions to goroutines leaks are grpc connections that are related to the storage cleanup, there is also a TODO in the code and an open issue I've also tried to reduce the number of connections between storage and etcd, right now, each Storage creates an independent conection to etcd, but sharing the etcd client doesn't work
I think I made some progress with it. I was playing with ttlcontroller test (as it's fairly simple one for the playground and I'm familiar with it, but symptoms in many tests are exactly the same) In I actually managed to reduce amount of leaked goroutines from 1023 to 85.

It's not mergable in this state, but it clearly shows that destroy functions for storage are not called in integration tests. I will try to dig deeper into those and clean this up - but I think the above clearly shows where the problem is. - FYI ^^
is out for review and it's making most of the job here There are still leftovers, but those are mostly oneof's here and there - we should address them independently.
And a bit more longerterm cleaner approach is in
With we're down to 14. I admit that this is one of the simplest tests, but apiserver shutdown is pretty clean once that PR merges.
/honk
! <detailsIn response to : /honk Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
a leftover for the storage cleanup in
Re comment above - yeah, I'm aware, we need to clean it up. FTR:
awesome, we are several order of magnitud better
Looking further into it, in different tests I'm seeing a lot of goroutines with the top of the stacktrace like this: should hopefully address those once it merges... ^^
OK - I think I have now the biggest thing contributing factor to the remaining tests, although I don't yet fully understand it and don't know how exactly to fix it. The main problem is basically that TestServer is not finishing its shutdown: So what exactly is happening is: we run TryDown the TestServer there: this underneath is triggering run of PreShutdown hooks: in the meantime ShutdownInitated signal is signaled: as part of running preshutdown hooks, the bootstrap controller is stopping endpoint reconciler: and then as part of doReconcile, the GET requests hangs forever: By adding time.Sleep(5s) (didn't try smaller values TBH) before the ShutdownInitiated is signalled, the TestServer is shutting clearly: Obviously that's not a solution, but I'm wondering why the GET request mentioned above is hanging forever when ShutdownInitiate is signaled and what we can do with this. - thoughts? [The easiest repro is to run: Adding this sleep basically reduces the number of leaked routines from O(1000) to O(20).]

<--- UPDATE ---In fact, setting ShutdownDelayDuration to 5s has the same effect. So there are two main questions: if setting ShutdownDelayDuration is reasonable why the request actually hangs forever instead of finishing with some error... <--- UPDATE ---I think there must have been something different, as I'm no longer able to reproduce the improvement with my current code. Will try to dig deeper. <-- UPDATE ---OK - the last commit in brings it back to fixed.
I thought that the reconciler hangs because of
Not in this place - here the usual GET request hangs - I will keep looking over the next days.
OK - I understand now and the above isn't really true. The request wasn't really hanging forever. What was happening is that it was being rejected by the WaitGroup handler: and retried by the client-go internals... So I guess from that perspective setting non-zero ShutdownDelayDuration actually makes sense.. - thoughts? ^^
mhm, not sure why setting should help. In general, the server shout stop listening and the HTTP std library should instruct the client's stack to tear down the underlying TCP connection. In that case, the client-go should fail on dialing and stop retrying releasing the server. thoughts?
It's that with ShutdownDelay, the "stopCh" is signaling PreShutdown hooks (that still want to do some work, including API calls) and if we immediately start rejecting that, they will not be able to do that work...
Discussed a bit more offline - I'm going to investigate blocking closing the waitGroup filter on preshutdown hooks. Will try to send a PR in upcoming days.
We're getting there - this is where we are now: I have a bit more in my local branch to upstream, but I need to clean them up and may not have time in the next week or so.
where are we standing today?
Getting close:
A bit more coming soon.
This is where we are now: I understand some of them (though fixing is potentially non-trivial) and need to understand the others (in particular serving). And we should prevent regressions by failing if we have more leaking goroutines - will be looking at that over upcoming weeks.

OK - so here is where we are (with the current check set to not leaking more than 4 goroutines): Comparing to the first comment in this issue - that looks like a significant improvement :) Here is the summary of I think the remaining goroutines in the above files: [x] integration/apiserver/tracing: The goroutines are coming from otel exporter (in particular grpc layer). It seems that something is not closed, although required digging deeper into otel exporter. [ ] integration/serving: The goroutines are coming from the failure case: And they basically are coming from the fact that Queue when created is spawning a goroutine: which is only shutdown on Shutdown() call. But Shutdown in all controllers is only called in defer of controllers of the function that is actually not being run in failure cases. The actual queue that is problematic here is coming from here: [x] testing/integration/controlplane/transformation The goroutines are coming from the fact that we're not closing the kms transformer (grpc goroutines) that is being created. We basically create a grpc service and never close that: Fixing would probably require changing transformer interfact to implement Close method or sth like that.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
- can you please help with the apiserver/tracing test mentioned in
looking
Opened to try fixing the goroutine leak.
It seems the fix worked fine - I'm marking the tracing as fixed - thanks I'm also trying to fix couple other smaller things.
I've opened as a final step. With that PR we're in good enough shape that I'm going to close this issue as further improvements would have negligible impact.


Question: readonly KUBE_CLIENT_PLATFORMS elif [[ "${KUBE_FASTBUILD:-}" == "true" ]]; then <del> readonly KUBE_SERVER_PLATFORMS=(linux/amd64) </del> <ins> KUBE_SERVER_PLATFORMS=(linux/amd64) readonly KUBE_SERVER_PLATFORMS </ins> readonly KUBE_NODE_PLATFORMS=(linux/amd64) if [[ "${KUBE_BUILDER_OS:-}" == "darwin"* ]]; then readonly KUBE_TEST_PLATFORMS=( darwin/amd64 linux/amd64 ) <del> readonly KUBE_CLIENT_PLATFORMS=( </del> <ins> KUBE_CLIENT_PLATFORMS=( </ins> darwin/amd64 linux/amd64 <del> ) </del> <ins> ) readonly KUBE_CLIENT_PLATFORMS </ins> else readonly KUBE_TEST_PLATFORMS=(linux/amd64) <del> readonly KUBE_CLIENT_PLATFORMS=(linux/amd64) </del> <ins> KUBE_CLIENT_PLATFORMS=(linux/amd64) readonly KUBE_CLIENT_PLATFORMS </ins> fi else KUBE_SERVER_PLATFORMS=("${KUBE_SUPPORTED_SERVER_PLATFORMS[@]}")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: building Kubernetes release and it failed due to a variable not being set (). It appears a does not function as expected on (used Ubuntu 16). Additional details will be provided in comment below What you expected to happen: successful build of Kuberentes release or documented minimum supported OS releases (i.e., bash versions) How to reproduce it (as minimally and precisely as possible): using Ubuntu 16 (), run [] using , or (containing or subsequent cherry pick) Anything else we need to know?: It appears the issue is with how and create variables, where I believe does not create the variables with the expected scope (created using scope versus scope). This creates a minimum supported OS version when running from the best I can tell. Environment: Kubernetes version (use ):  Cloud provider or hardware configuration: N/A OS (e.g: ):Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): N/A Others: /sig release
Some of the details I mentioned above, modified the file to output some helpful information the diff of changes I made the results when running on Ubuntu 16 (bash 4.3), the variable appears to be scoped inside the function the results when running on Ubuntu 16 (bash 4.3) without the variable attribute (you can now see the variable is a variable, not locally scoped to the function ) And on Ubuntu 18 (bash 4.4) the variable is scoped with the attribute as you would expect and runs as expected.
I guess fixes this problem. cc
I don't think the is/was the only issue here. It was obviously as I was testing with , so only variables were being used and not scoped outside the function (in bash 4.3).
we are also getting the same error on bash version 4.3 cc
any update on this, are you fixing it??
Same question with , this issue impact OpenLab cloud-provider-openstack E2E testing job.
can you please try this?

it works with your fix , previously jobs were failing --- EDIT --- Hold on, I think I was too fast, it failed in a later stage don't know if is a problem or related to the script changes. Should we have the tar files as an output ?
Sorry I missed this yesterday. I will try as well. Otherwise, my assumption was to revert the use of a function, as that creates the scoping problem we've been seeing with variables in .
i think that would be not
Also both not should work now.
Looks like worked for me. Was able to run a using successfully.
ref: as the pr that appears to have caused this


Question: // Update replaces an existing event. Returns the copy of the event the server returns, or an error. func (c *FakeEvents) UpdateWithEventNamespace(event *v1.Event) (*v1.Event, error) { <del> action := core.NewRootUpdateAction(eventsResource, event) </del> <ins> var action core.UpdateActionImpl </ins> if c.ns != "" { action = core.NewUpdateAction(eventsResource, c.ns, event) <ins> } else { action = core.NewUpdateAction(eventsResource, event.GetNamespace(), event) </ins> } obj, err := c.Fake.Invokes(action, event) if obj == nil {

Opening this issue based on what I ran into on with namespace as follows While unit testing the controller, we create a fake client and setup the controller to run using the client. This ends up creating a having the namespace set to When the call eventually reaches The action gets associated with a namespace only if the object has a namespace associated with it. Which in this will be Which will result in the error in to manage the event like does. In case if the namespace is set to Above is a test case that can easily simulate the issue in question. Referring back to the test under PR NA <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: test: "Pod with an Affinity matches node's labels but the PodSpec.NodeSelector(the old thing that we are deprecating) " + "is not satisfied, won't schedule onto the node", }, <ins> { pod: &v1.Pod{ Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "foo", Operator: v1.NodeSelectorOpNotIn, Values:   []string{"invalid value: ___@#$%^"}, }, }, }, }, }, }, }, }, }, labels: map[string]string{ "foo": "bar", }, fits: false, test: "Pod with an invalid value in Affinity term won't be scheduled onto the node", }, </ins> } expectedFailureReasons := []algorithm.PredicateFailureReason{ErrNodeSelectorNotMatch}

SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected Has been failing consistently since 4/17. Here are the /kind bug /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 cc /assign for triage.
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : ## Failing Job SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.11
/status approved for milestone
Also fails as part of upgrade job as seen below
Also fails as part of following jobs https://k8s- https://k8s- https://k8s- https://k8s-
xref
/assign
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 6 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Seems like this is failing in (just with slightly different versions)


Question: // Can't check exact strings due to terminal control commands (colors) requiredItems := []string{"Kubernetes master", "is running at"} if framework.ProviderIs("gce", "gke") { <del> requiredItems = append(requiredItems, "KubeDNS", "Heapster") </del> <ins> requiredItems = append(requiredItems, "Heapster") </ins> } for _, item := range requiredItems { if !strings.Contains(output, item) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: [sig-cli] Kubectl client [] Kubectl cluster-info should check if Kubernetes master services is included in cluster-info [Conformance] 10s e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-cli]sKubectlsclients[k8s.io]sKubectlscluster-infosshouldschecksifsKubernetessmastersservicessissincludedsinscluster-infoss[Conformance]$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig cli