Question: # Please see http://docs.fluentd.org/articles/install-by-deb for more # information about installing fluentd using deb package. <del> FROM ubuntu:14.04 </del> <ins> FROM gcr.io/google_containers/ubuntu-slim:0.4 </ins> MAINTAINER Alex Robinson "arob@google.com" MAINTAINER Jimmi Dyson "jimmidyson@gmail.com"

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: var _ = SIGDescribe("Events", func() { f := framework.NewDefaultFramework("events") <del> /* Release: v1.9 Testname: Pod events, verify event from Scheduler and Kubelet Description: Create a Pod, make sure that the Pod can be queried. Create a event selector for the kind=Pod and the source is the Scheduler. List of the events MUST be at least one. Create a event selector for kind=Pod and the source is the Kubelet. List of the events MUST be at least one. Both Scheduler and Kubelet MUST send events when scheduling and running a Pod. */ framework.ConformanceIt("should be sent by kubelets and the scheduler about pods scheduling and running ", func() { </del> <ins> ginkgo.It("should be sent by kubelets and the scheduler about pods scheduling and running ", func() { </ins> podClient := f.ClientSet.CoreV1().Pods(f.Namespace.Name)

Related: Noticed this as part of Opened in response to: According to our , the e2e test is ineligible for Conformance. Specifically, the test in question is looking for guaranteed delivery of Events from specific sources (kubelet and default scheduler). This cannot be reasonably expected of all Kubernetes clusters, and thus the test should be demoted from Conformance. For historical context: It looks like this has been part of Conformance sometime since sometime prior to 2015-10-17 (we introduced the [Conformance] tag in 1.2, ref: We overhauled Conformance testing policy (including exclusion of Events) on 2018-10-08 (ref: ) If I wanted to be a language lawyer about this, I could say the policy covers ineligibility for promotion but said nothing about the existing corpus of tests. But I think this was just an honest oversight when we reviewed the existing Conformance tests. Due to the very nature of what this test is verifying, I don't think it's possible to rewrite to avoid Events. Thus, I think it should be demoted from Conformance. SIG Node is free to continue to treat this as release-blocking, but it's not something we can reasonably expect of any Kubernetes cluster /kind cleanup /priority important-soon /milestone v1.23 /sig architecture /area conformance because conformance /sig node because it's a sig-node test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Opened to address FYI Since y'all were commenting on
+1


Question: } // if this pod is being synced for the first time, we need to make sure it is an active pod if options.Pod != nil && (options.Pod.Status.Phase == v1.PodFailed || options.Pod.Status.Phase == v1.PodSucceeded) { <del> // check to see if the pod is not running and the pod is terminal. // If this succeeds then record in the podWorker that it is terminated. </del> <ins> // Check to see if the pod is not running and the pod is terminal; if this succeeds then record in the podWorker that it is terminated. // This is needed because after a kubelet restart, we need to ensure terminal pods will NOT be considered active in Pod Admission. See http://issues.k8s.io/105523 // However, `filterOutInactivePods`, considers pods that are actively terminating as active. As a result, `IsPodKnownTerminated()` needs to return true and thus `terminatedAt` needs to be set. </ins> if statusCache, err := p.podCache.Get(uid); err == nil { if isPodStatusCacheTerminal(statusCache) { <ins> // At this point we know: // (1) The pod is terminal based on the config source. // (2) The pod is terminal based on the runtime cache. // This implies that this pod had already completed `SyncTerminatingPod` sometime in the past. The pod is likely being synced for the first time due to a kubelet restart. // These pods need to complete SyncTerminatedPod to ensure that all resources are cleaned and that the status manager makes the final status updates for the pod. // As a result, set finished: false, to ensure a Terminated event will be sent and `SyncTerminatedPod` will run. </ins> status = &podSyncStatus{ terminatedAt:       now, terminatingAt:      now, syncedAt:           now, startedTerminating: true, <del> finished:           true, </del> <ins> finished:           false, </ins> fullname:           kubecontainer.BuildPodFullName(name, ns), } }

New TestGrid alerts at since commit/. Found 2 failing rows in node-kubelet-serial-containerd: 2 new, 0 other 2 new Test Failures in node-kubelet-serial-containerd. Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Not sure suspect No response /sig node release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake /remove-kind failing-test See
/remove-sig release
Taking a look at The test failed due to: so it means that the pod was not terminated and deleted properly in the test that ran prior. The is the one creating this pod, but I would expect that as part of the test teardown, the namespace is deleted (and thus the pod should be deleted).
The services log show that the kubelet never sent the request to delete the pod, that is strange...
From the build-logs, the test execution for when the pod was created: Kubelet logs for the pod - Full logs are here -
Here's the termination part of the logs:
It looks like to me the issue here is that the kubelet is being restarted in a bit of an unfortunate time resulting in at least two related issues: The pod status has been updated on the api-server to terminal just prior to the kubelet restarting. The status manager did not yet complete . Then after kubelet restarts and comes back up, it attempts to add this pod , (even though it is in terminal phase). This seems like the first issue: if the pod phase is terminal on the API server, we probably should not add it back.

The pod worker receives the , but the pod worker quickly terminates (). I am assuming this is because of , i.e. based on pod cache lookup the pod is terminal. The side affect of this is that the status manager never receives the signal that the pod is (which is normally done as part of . This is because in this case, we never call which sets to be true. As a result, I think will never return true, so this pod will never be able to be deleted.
I have made a test which will reproduce this behavior in a node e2e, see It can be reproduced by starting a pod and then deleting it and triggering kubelet restart at the right time. A watch is setup to trigger a kubelet restart as soon as the pod enters the succeeded phase. If the kubelet restart happens before the kubelet is able to delete the pod on the API server, but after the phase is updated, then we trigger the issue. The pod will be re- to kubelet after restart, but will never be deleted on the API server despite it having a deletion timestamp set and being in terminal phase. Here are logs from the and running the above repro node e2e test. Here is the kubelet logs portion prior to the restart: and after restart: It also looks like we are hitting another invalid state in based on the log assertion: I tried the same test on 1.26 and it passes. This is because on 1.26, the status manager phase to terminal happens at roughly at the same time when we delete the pod. On 1.27+, after , the status phase to terminal can happen earlier and the status manager can skip deleting the pod because . In this case, after start, will never be true, so the pod never will get deleted.
But in general pods can be in terminal phase and not be deleted (like plain pods and pods from jobs). They would eventually be deleted by the pod GC otherwise. Is this a bug in the device plugin?
That is true, but the issue here is that the pods have a deletion timestamp set as well. If the deletion timestamp is set and the kubelet is up, it is the kubelet's responsibility to terminate and delete those pods.

This doesn't have anything with device plugin, rather we caught the issue due to the device plugin test which asserts that all pods are terminated and deleted prior to starting the test.
I have a WIP fix for this issue in
That makes sense
Should we aim 1.27 with this fix? This seems like an important regression.
I discussed this with yesterday and we agreed that this can wait until 1.27.1 Clients should be able to handle terminated pods. Worst case scenario, the pod GC would take care of these pods eventually.
Cherrypicked to v1.27 (to be released in 1.27.2) in


Question: "k8s.io/apimachinery/pkg/util/sets" ) <del> // ciphers maps strings into tls package cipher constants in // https://golang.org/pkg/crypto/tls/#pkg-constants // to be replaced by tls.CipherSuites() when the project migrates to go1.14. var ciphers = map[string]uint16{ "TLS_RSA_WITH_3DES_EDE_CBC_SHA":           tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA, "TLS_RSA_WITH_AES_128_CBC_SHA":            tls.TLS_RSA_WITH_AES_128_CBC_SHA, "TLS_RSA_WITH_AES_256_CBC_SHA":            tls.TLS_RSA_WITH_AES_256_CBC_SHA, "TLS_RSA_WITH_AES_128_GCM_SHA256":         tls.TLS_RSA_WITH_AES_128_GCM_SHA256, "TLS_RSA_WITH_AES_256_GCM_SHA384":         tls.TLS_RSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, "TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA":     tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384":   tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384": tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305":    tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305":  tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, "TLS_AES_128_GCM_SHA256":                  tls.TLS_AES_128_GCM_SHA256, "TLS_CHACHA20_POLY1305_SHA256":            tls.TLS_CHACHA20_POLY1305_SHA256, "TLS_AES_256_GCM_SHA384":                  tls.TLS_AES_256_GCM_SHA384, } </del> <ins> var ( // ciphers maps strings into tls package cipher constants in // https://golang.org/pkg/crypto/tls/#pkg-constants ciphers         = map[string]uint16{} insecureCiphers = map[string]uint16{} ) </ins> <del> // to be replaced by tls.InsecureCipherSuites() when the project migrates to go1.14. var insecureCiphers = map[string]uint16{ "TLS_RSA_WITH_RC4_128_SHA":                tls.TLS_RSA_WITH_RC4_128_SHA, "TLS_RSA_WITH_AES_128_CBC_SHA256":         tls.TLS_RSA_WITH_AES_128_CBC_SHA256, "TLS_ECDHE_ECDSA_WITH_RC4_128_SHA":        tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, "TLS_ECDHE_RSA_WITH_RC4_128_SHA":          tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA, "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, </del> <ins> func init() { for _, suite := range tls.CipherSuites() { ciphers[suite.Name] = suite.ID } // keep legacy names for backward compatibility ciphers["TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"] = tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 ciphers["TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305"] = tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 for _, suite := range tls.InsecureCipherSuites() { insecureCiphers[suite.Name] = suite.ID } </ins> } // InsecureTLSCiphers returns the cipher suites implemented by crypto/tls which have

Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,


Question: awaiting <-chan interface{} finished *time.Time lock     sync.Mutex <del> notify   chan struct{} </del> <ins> notify   chan bool </ins> } // Operations tracks all the ongoing operations.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue.

<detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.


Question: visibility = ["//visibility:public"], deps = [ "//pkg/api/v1/pod:go_default_library", <del> "//pkg/apis/apps:go_default_library", </del> "//pkg/apis/core:go_default_library", <ins> "//pkg/apis/extensions:go_default_library", </ins> "//pkg/apis/scheduling:go_default_library", "//pkg/kubelet/apis:go_default_library", "//pkg/quota/evaluator/core:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug We can see all related with test failed. and so on. I think it maybe relate with PR and . What happened: Always fails. What you expected to happen: Pass all.
/cc PTAL. Thanks.
Why did the not pass the test but was still merged?
/priority important-soon /kind e2e-test-failure
/area platform/gce
Looks like the fix is in
is not a blocking / required test (for merge) currently IIRC, kubernetes/kubernetes uses the "submit-queue" which will only pay attention to some tests.
i think this is true, as confirmed by the current state of this PR: it fails the test, but the CI says instead of is there are way to see which tests would cause a ?
off the top of my head the definitive way is this list: but note that is working on moving us to in the nearish future, and every other kubernetes repo besides this one is either not using merge automation or is using , which behaves differently and has seperate config integrated with the rest of test-infra.
thanks for the details! :+1:


Question: go func() { err, _ := <-errOnce.Err() if err != nil { <del> t.Fatalf("delegate %d: unexpected error: %v", i, err) </del> <ins> t.Errorf("delegate %d: unexpected error: %v", i, err) </ins> } }() }

Can we get this fixed ASAP? Thanks!
yep, thanks for reporting , Brendan Burns wrote:
how reproduceable is this? I have some changes that I think will resolve it but I couldn't reproduce locally even when all my cpu's were pretty tied up and using a high GOMAXPROCS.
flakey label process?
the labelling is due to me failing @ github's label interface. Bulk editing labels removes them if any of the selected things have the label sigh Thanks for the quick fix.


Question: defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CSIMigrationGCE, tc.csiMigration)() defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InTreePluginGCEUnregister, tc.csiMigration)() <del> plugins = gcepd.ProbeVolumePlugins() </del> <ins> // if InTreePluginGCEUnregister is enabled, only the CSI plugin is registered but not the in-tree one </ins> plugins = append(plugins, csi.ProbeVolumePlugins()...) } else { plugins = controllervolumetesting.CreateTestPlugin()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g.

because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted


Question: defaultBinderPlugin framework.BindPlugin } <del> func newFirstFailBindPlugin(_ runtime.Object, handle framework.Handle) (framework.Plugin, error) { binder, err := defaultbinder.New(nil, handle) if err != nil { return nil, err } return &firstFailBindPlugin{ defaultBinderPlugin: binder.(framework.BindPlugin), }, nil } </del> func (*firstFailBindPlugin) Name() string { return "firstFailBindPlugin" }

pull-kubernetes-integration: scheduler TestRequeueByBindFailure Running with stress reproduces the flake Not sure, seeing a lot of retests on pull-kubernetes-integration Bind logic in scheduler integration tests just changed 3 days ago in , not sure if that's related No response No response /sig scheduling /priority important-soon cc
I doesn't look related to the latest change you were also looking at Bind. Any ideas?
It's a little strange because the error reports but since the bind-plugin will fail for the first calling time and there's no kubelet, that means the pod status will be kept since. I created another PR to help output the error message, which maybe helpful for analysis.
This test is very straightforward: the first try to bind "pod-1", the Pod is rejected by the binding plugin. () scheduling queue puts back this "pod-1" to activeQ/backoffQ directly, and "pod-1" will soon come back to the scheduling cycle. the second try to bind "pod-1", it's successful. And, means that (1) doesn't reject "pod-1" correctly. And I don't have any idea why it can happen. The only scenario I can think of is that there is somehow another Pod created before pod-1. When is executed , is it possible that any other Pods (like system ones) are created automatically?
/triage accepted current top flake of integration
/assign I've got one possible scenario, will have a quick fix.


Question: if secureLocation != "" { secureServer := &http.Server{ Addr:           secureLocation, <del> Handler:        apiserver.RecoverPanics(m.Handler), </del> <ins> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(m.Handler)), </ins> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, MaxHeaderBytes: 1 << 20,

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)


Question: func (cmo createNamespacesOp) patchParams(w *workload) (realOp, error) { if cmo.CountParam != "" { <del> var ok bool if cmo.Count, ok = w.Params[cmo.CountParam[1:]]; !ok { return nil, fmt.Errorf("parameter %s is undefined", cmo.CountParam) </del> <ins> var err error cmo.Count, err = w.Params.get(cmo.CountParam[1:]) if err != nil { return nil, err </ins> } } return &cmo, (&cmo).isValid(false)

Note: this issue should be viewed in the context of . If this happens, this is very likely a typo where someone used a parameter in a way they didn't mean to (for example, ). /sig scheduling /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi this is my first time contributing to kubernetes. can you provide me some guidance on how to to get started?
Hi, is this issue still valid to work on?
yes, this is still valid. I recommend starting with the and reading from there.
are you still working on this issue or should I work on it.
I would like to work on this issue if you haven't started already.
would it be ok if I took a look at this?
/assign
Hi Can you help me in why its not assigning the values to createNodes and createPods directly and using the opcode and countParam in performance-
We use a field to differentiate between different operations. The value of the field (e.g. "createNodes") determines the operation. The field defines the number of pods/nodes to create. What do you mean by "assigning the values to createNodes and createPods directly"?
Thanks for the information.

By assigning, I just meant that whether we can assign createNodes directly like ( createNodes: $initNodes ).
I don't think that's possible since there can potentially be other parameters besides the countParam. For example: Different operations could potentially have different parameters.
createNodes and createPods aren't the only operations. Other operations (e.g. churn and barrier), including future ones, may not fit that format. It's also easier to parse and more explicit this way.
Thanks , Having a doubt more, in performance- under the name:SchedulingPodAntiAffinity, ! why are we giving the same values in 5000Nodes as given in 500Nodes, or this is an error.
that sounds like a typo indeed
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello Are you still going to work on this issue?
Hi I have worked on the issue but not working for a while as was unsure about the exact problem.
, no problem if you are willing to work on this issue, shall unassign myself
/unassign
Thanks :)
/assign
/remove-lifecycle stale
hey! I am beginner and I interested in scheduling and I know basics of scheduling , so how can I start to contribute on this issue.
Sorry, PR for this issue has already created.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: // Unmount removes the mount reference from the volume allowing it to be // re-mounted elsewhere. func (u *storageosUtil) UnmountVolume(b *storageosUnmounter) error { <ins> klog.V(4).Infof("clearing StorageOS mount reference for volume %q with namespace %q", b.volName, b.volNamespace) </ins> if err := u.NewAPI(b.apiCfg); err != nil { // We can't always get the config we need, so allow the unmount to // succeed even if we can't remove the mount reference from the API. <del> klog.V(4).Infof("Could not remove mount reference in the StorageOS API as no credentials available to the unmount operation") </del> <ins> klog.Warningf("could not remove mount reference in the StorageOS API as no credentials available to the unmount operation") </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a PVC is created and the PV mounted into a Pod using the StorageOS in-tree volume driver, the StorageOS CLI & UI do not report that the volume is mounted. The PV is created and mounted into the Pod successfully, and all k8s functionality works as expected. However, since the correct mount status is not visible to administrators using the StorageOS UI or CLI, they may make changes to or delete a mounted volume without realising, and the safeguards (must use to delete a mounted volume) do not trigger. This is a regression caused by , released in 1.14.0. What you expected to happen: The StorageOS CLI should report volume mount information (node name, mount timestamp). How to reproduce it (as minimally and precisely as possible): StorageOS node containers with CSI disabled: application Pod with a PVC using the StorageOS StorageClass: StorageOS to list volumes: Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 and later Cloud provider or hardware configuration: any OS (e.g: ): any linux (incl. Ubuntu 18.04) Kernel (e.g. ): any linux (incl. 4.15.0-33-generic) Install tools: any (incl. minikube) Network plugin and version (if this is a network-related bug): N/A Others: Requires StorageOS deployment /sig storage /assign


Question: // that do not specify any priority class and there is no priority class // marked as default. DefaultPriorityWhenNoDefaultClassExists = 0 <ins> // SystemPriorityClassPrefix is the prefix reserved for system priority class names. Other priority // classes are not allowed to start with this prefix. SystemPriorityClassPrefix = "system-" </ins> ) // +genclient

In a recent discussion on , suggested to keep priority class names that start with "system-" for system use only and not allow user defined priority classes to start with this prefix. Given that we have reserved priority values 1 Billion for system use only, it makes sense to reserve a prefix as well to ensure that predefined priority classes won't conflict with user defined ones in the future. Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig scheduling /assign ref


Question: kubeclientset "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5" "k8s.io/kubernetes/pkg/client/record" "k8s.io/kubernetes/pkg/controller" <ins> "k8s.io/kubernetes/pkg/conversion" </ins> "k8s.io/kubernetes/pkg/runtime" "k8s.io/kubernetes/pkg/util/flowcontrol" "k8s.io/kubernetes/pkg/watch"

Unit tests on PR are failing due to data race in federation-controller/namespace unittest. More logs: https://k8s-
CC
Sent to fix this


Question: "audit.go", "certificates.go", "framework.go", <ins> "node_authz.go", </ins> "service_accounts.go", ], tags = ["automanaged"],

Is this a BUG REPORT or FEATURE REQUEST? (choose one): TESTING REQUEST Spawned from: It would be nice with some e2e tests for the Node Authorizer so consumers of it (kubeadm, GCE, GKE, etc.) could easily make sure it's enabled and working, just by editing when running e2e's. I wrote a quick bash script of what it could do: The bash script above is in no way comprehensive enough, but on the other hand it took ~5mins to write :) For now (1.7), we might use something like this bash script in our kubeadm e2e CI job to make sure the Node Authorizer/Admission plugin is enabled correctly. cc
was assigned for triaging this... Does such an e2e test already exist somewhere? said that he could take this on if not...
no negative e2e test exists (make sure things the Node authorizer disallows fail) negative integration tests exist at negative e2e tests for this will be tricky, since you'll need node credentials and a specific authz and admission config.
See the above bash script, I was able to do it purely from kubectl by impersonating the group and node user... If we feature-gate this e2e test, we can run it for the deployments that actually have unique node credentials
cc this maybe could be a good task for you... :)
Cool, thanks for point me this.
You basically only have to implement these commands as an e2e test initially (by impersonating a node in the cluster). Then when that's done, we can check if there are more than these four cases that needs to be verified. This should go under a separate and be enabled in deployments (like kubeadm) where nodes have unique credentials.
I didn't find a way to simulate a node credential, do you have any good idea ? after reading the integration testing code, I fell it's enough the guarantee the quality, WDYT?


Question: // DefaultEvictionHard includes default options for hard eviction. var DefaultEvictionHard = map[string]string{ <del> "memory.available":  "100Mi", "nodefs.available":  "10%", "nodefs.inodesFree": "5%", "imagefs.available": "15%", </del> <ins> "memory.available":   "100Mi", "nodefs.available":   "10%", "nodefs.inodesFree":  "5%", "imagefs.available":  "15%", "imagefs.inodesFree": "5%", </ins> }

It is possible to deploy Kubernetes with a separate filesystem. We default the settings for EvictionHard . A concern is that if users are using the default settings and separate filesystem for images, we are effectively turning off monitoring for INodes on the imagefs. This means that we won't have disk pressure if inodes run out on the ImageFs. The fix can be to add To default settings. This doesn't seem to be a regression as these defaults were 5 years ago (k8s 1.12) but this could be a potential bug if either users mount a separate filesystem to their container runtime storage or they configure their runtime to store their images on a separate filesystem.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
/kind feature


Question: translateTimestamp(e.FirstTimestamp), translateTimestamp(e.LastTimestamp), e.Count, <del> e.Source, </del> <ins> formatEventSource(e.Source), </ins> e.InvolvedObject.FieldPath, e.Type, e.Reason,

Greetings, currently, when doing then in column, the ui result is maybe can do something like this instead? remove then join it with
How about something like ?
Greetings, thanks for the interest, I thought I may throw hacking on it. btw, how about for ?
For we can just print since it's master's component.
Hi looks like there is the same issue reported on the OpenShift Origin github: It would make more sense to solve that upstream. Should we do something together to solve it?
There is a pull request for that:


Question: Fail(fmt.Sprintf("Internal error: invalid parameters passed to deleteServiceOrFail: clientset: %v, namespace: %v, service: %v", clientset, namespace, serviceName)) } framework.Logf("Deleting service %q in namespace %v", serviceName, namespace) <ins> err := deleteService(clientset, namespace, serviceName, orphanDependents) if err != nil { framework.ExpectNoError(err, "Error deleting service %q from namespace %q", serviceName, namespace) } } func deleteService(clientset *fedclientset.Clientset, namespace string, serviceName string, orphanDependents *bool) error { </ins> err := clientset.CoreV1().Services(namespace).Delete(serviceName, &metav1.DeleteOptions{OrphanDependents: orphanDependents}) <del> framework.ExpectNoError(err, "Error deleting service %q from namespace %q", serviceName, namespace) </del> <ins> if err != nil { return err } </ins> // Wait for the service to be deleted. err = wait.Poll(5*time.Second, fedframework.FederatedDefaultTestTimeout, func() (bool, error) { _, err := clientset.Core().Services(namespace).Get(serviceName, metav1.GetOptions{})

In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


Question: require.NoError(t, err) assert.EqualValues(t, tc.nameservers, ns, "test case [%d]: name servers", i) assert.EqualValues(t, tc.searches, srch, "test case [%d] searches", i) <del> assert.EqualValues(t, tc.options, opts, "test case [%d] options", i) </del> <ins> assert.EqualValues(t, sets.NewString(tc.options...), sets.NewString(opts...), "test case [%d] options", i) </ins> } else { require.Error(t, err, "tc.searches %v", tc.searches) }

If the content of is only could recognize the last option and is ignored. And user has to use to workaround the issue. But this format is valid actually and able to read correctly in golang and libc. Able to parse multiple options lines in correctly. Use in the host's file and then inside POD's , it would become No response Any version. N.A. Ubuntu 22.04 No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-node </details
/sig node
Seems this behaviour is introduced by you at , is it an expected behaviour or a bug?
are those options parsed and used correctly or one of both is ignored?
Others are ignored and online the last line is parsed by kubelet correctly.
and in a normal , does libc use all the options or ignore some and use only one?
I tried centos/ubuntu, both will append extra options to the options list and overwrite them if it already exists by the later settings. For instance, is the same as . But I still think the workaround looks a perfered format.
Yes, but it looks like a bug since kubelet's behavior is different.
I tried to fix it in But I am still not sure if this is a valid fix. I will wait for pr review comments or further confirmation about whether this is valid.
/sig network
EDITED The feature was by a long time ago by There is a design kubernetes/community. The PR does not follow the design per The KEP can be found in KEP
/remove-sig node
Previous discussion and interesting comment from
Just want to clarify that these two are not related --- the change does not follow the design.
Just have the book on my desktop. !
I think the book just said could write it as , but didn't mention is invalid.

And since glibc/golang DNS resolve is able to recognize it correctly, should we consider it as a de-facto way that we should support?
what about muslc, does it support it? we have to support muslc too, a lot of busybox/alpine users can break
I did some initial test with busybox:1.34.1-musl. It supports multi options. for , it works with multi-options. ndots:1, ping will fail for .
/assign
I'm trying to understand this, how the options are merged if they are duplicated? the man page for is clear ,notation%20as%20per%20RFC%. for only says it seems that apply the same logic to handle conflict if I try with this only the last option is used (same for muslc and libc) if I have two different options in different lines then things start to get more complicated, because I can not test all possible options combinations and musl resolver is not compatible with glibc resolver , see I'm worried we can break musl based containers by assuming the host configuration, that is mostly libc based, will be compatible with the container. I think we should be conservative and document this behavior of the dns options, and ask users to merge the options in the same line, raise a warning of something too EDIT I see is suggesting to parse all options and merge them in one line :eyes:
Add a small warning in log maybe another option. We may document it with some details. This is a little aggressive way.
I still can't find any spec that says how this is SUPPOSED to work, and I share concerns that this is likely to break someone somewhere. Did someone encounter this in the wild? Does it REALLY make sense to copy ALL of the options from the host config? Does it make sense to copy ANY of them? Looking at the man page, they seem very specific and maybe we should just leave them alone. What does Docker do here?
I found that there is a logic to merging DNS options in the pod DNS spec generating process. So I followed the logic. If the pod is using with , the current kubelet will merge the node's last option line in with . I did a simple test in Centos 7.9 with docker 19.03.8 like below.

keeps the options as is if no new DNS options. will overwrite the local DNS options. there are multi , docker will not merge them and keep them in one line.
Absent a real user who is really impacted by this, my inclination is to (again) do nothing. The risk of breakage is non-zero. The reward is zero (nobody is complaining, IIUC).
Hi, actually I am impacted by this (in production services) . And looks like I am not the first user who raise this, as other issues link to this thread.
Thanks for your input, I checked muslc and verified via code, muslc would parse all options if we write multiple line . So I don't think would break any user. Instead, it addresses the similar problem happens to muslc users as well.
OK. I am accepting it. Since libc's seem to do this anyway, I don't see why we should break it.
Many thanks


Question: t.Fatalf("expected %v ports but got %v:n%v", expected, actual, string(b)) } } <ins> // TestApplyCRDUnhandledSchema tests that when a CRD has a schema that kube-openapi ToProtoModels cannot handle correctly, // apply falls back to non-schema behavior func TestApplyCRDUnhandledSchema(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.ServerSideApply, true)() server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } noxuDefinition := fixtures.NewNoxuCustomResourceDefinition(apiextensionsv1beta1.ClusterScoped) // This is a schema that kube-openapi ToProtoModels does not handle correctly. // https://github.com/kubernetes/kubernetes/blob/38752f7f99869ed65fb44378360a517649dc2f83/vendor/k8s.io/kube-openapi/pkg/util/proto/document.go#L184 var c apiextensionsv1beta1.CustomResourceValidation err = json.Unmarshal([]byte(`{ "openAPIV3Schema": { "properties": { "TypeFooBar": { "type": "array" } } } }`), &c) if err != nil { t.Fatal(err) } noxuDefinition.Spec.Validation = &c noxuDefinition, err = fixtures.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } kind := noxuDefinition.Spec.Names.Kind apiVersion := noxuDefinition.Spec.Group + "/" + noxuDefinition.Spec.Version name := "mytest" rest := apiExtensionClient.Discovery().RESTClient() yamlBody := []byte(fmt.Sprintf(` apiVersion: %s kind: %s metadata: name: %s spec: replicas: 1`, apiVersion, kind, name)) result, err := rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw() if err != nil { t.Fatalf("failed to create custom resource with apply: %v:n%v", err, string(result)) } verifyReplicas(t, result, 1) // Patch object to change the number of replicas result, err = rest.Patch(types.MergePatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Body([]byte(`{"spec":{"replicas": 5}}`)). DoRaw() if err != nil { t.Fatalf("failed to update number of replicas with merge patch: %v:n%v", err, string(result)) } verifyReplicas(t, result, 5) // Re-apply, we should get conflicts now, since the number of replicas was changed. result, err = rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw() if err == nil { t.Fatalf("Expecting to get conflicts when applying object after updating replicas, got no error: %s", result) } status, ok := err.(*errors.StatusError) if !ok { t.Fatalf("Expecting to get conflicts as API error") } if len(status.Status().Details.Causes) != 1 { t.Fatalf("Expecting to get one conflict when applying object after updating replicas, got: %v", status.Status().Details.Causes) } // Re-apply with force, should work fine. result, err = rest.Patch(types.ApplyPatchType). AbsPath("/apis", noxuDefinition.Spec.Group, noxuDefinition.Spec.Version, noxuDefinition.Spec.Names.Plural). Name(name). Param("force", "true"). Param("fieldManager", "apply_test"). Body(yamlBody). DoRaw() if err != nil { t.Fatalf("failed to apply object with force after updating replicas: %v:n%v", err, string(result)) } verifyReplicas(t, result, 1) } </ins>

What happened: While upgrading from v1.15 to try out kubernetes v1.16.0-beta.2, kube-apiserver is printing error messages: This also brakes the API server since the user is unable to even delete the "faulty" CR What you expected to happen: Not to print any error message and accept a CR that was already valid in 1.15 How to reproduce it (as minimally and precisely as possible): Install k8s 1.15.x (optional) Create the following CR Upgrade to k8s v1.16.0-beta.2check kube-apiserver for errors:Try delete the faulty CR (Won't be possible) Environment: Kubernetes version (use ): Anything else we need to know?: A quick search sent me to this code path that is being used to compile k8s It seems the problem it known but not yet fixed since this is in the latest commit of : /kind bug /king regression cc
/sig api-machinery
/assign /priority critical-urgent /milestone v1.16
This is caused by compiling the openapi spec for use by server-side apply here: SSA is using the method, which has significant gaps in handling valid openapi specs (see all the TODO comments in ) and until now was only used for client-side kubectl validation (as far as I'm aware). Using it in the server-side path is concerning. /assign
running with , this works as expected and is compatible with 1.15
/assign
fix and covering test is in
this is resolved in , available now


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/fake:go_default_library", "//staging/src/k8s.io/cloud-provider/volume:go_default_library", "//vendor/github.com/aws/aws-sdk-go/aws:go_default_library", <ins> "//vendor/github.com/aws/aws-sdk-go/aws/awserr:go_default_library", </ins> "//vendor/github.com/aws/aws-sdk-go/service/ec2:go_default_library", "//vendor/github.com/aws/aws-sdk-go/service/elb:go_default_library", <ins> "//vendor/github.com/aws/aws-sdk-go/service/elbv2:go_default_library", </ins> "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/github.com/stretchr/testify/mock:go_default_library", "//vendor/github.com/stretchr/testify/require:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using a pod like the nginx-ing-cntlr with an AWS Network Load Balancer (NLB), and then draining the node its running on to force a migration to another node pool, does not reliably deregister the drained nodes from the NLB Target group. This causes the nodeport used by the NLB to remain open, and the drained node ends up continuing to funnel traffic for other pods cause of being used. This leads to connection timeouts for clients, and downtime on migrations. Switching to a Classic ELB, instead of an NLB, with connection draining enabled does work, and the instances that are drained are practically removed immediately from the instances targeted by the ELB, as expected. There seems to be some sort of issue with NLB's as this is not happening, or happening rarely (only saw it once in many tests but couldn't recreate it). The node + LB set seems to be working as I see in the controller-mgr logs that the nodes are updated as I went from 11 nodes to 8, after attempting to drain + migrate a nodepool of 3 nodes with nginx-ing-cntlr all running on them: However, this update does not seem to be translating into deregistering instances from the NLB. And i've waited minutes & hours in case timeouts were playing a hand, but that didn't seem to help. What you expected to happen: On node draining, with being used, that the NLB would deregister the instances as expected. How to reproduce it (as minimally and precisely as possible): Launch a Deployment + NLB service on a node and drain it to force it over to another node pool. Watch the instances in the AWS NLB Target group never get out of "healthy" status or be drained + removed. Anything else we need to know?: Environment: AWS EKS Kubernetes version (use ): 1.12 & 1.13 Cloud provider or hardware configuration: AWS OS (e.g: ): AWS Linux 2 Kernel (e.g.

): 4.14.123-111.109.amzn2.x8664 Install tools: Network plugin and version (if this is a network-related bug): core-dns v1.2.6 Others:
/sig aws
Related:
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if set.Status.Replicas != 3 { t.Error("Falied to scale statefulset to 3 replicas") } <del> pods, err := spc.addTerminatedPod(set, 3) </del> <ins> pods, err := spc.addTerminatingPod(set, 3) </ins> if err != nil { t.Error(err) } <del> pods, err = spc.addTerminatedPod(set, 4) </del> <ins> pods, err = spc.addTerminatingPod(set, 4) </ins> if err != nil { t.Error(err) }

/assign flakes in PR jobs more than once per day The can filter down to searching for results for a single test:
This test continues to flake PRs at least once per day:
https://k8s-
looking at it


Question: } } } <ins> // TestDeltaFIFO_PopShouldUnblockWhenClosed checks that any blocking Pop on an empty queue // should unblock and return after Close is called. func TestDeltaFIFO_PopShouldUnblockWhenClosed(t *testing.T) { f := NewDeltaFIFOWithOptions(DeltaFIFOOptions{ KeyFunction: testFifoObjectKeyFunc, KnownObjects: literalListerGetter(func() []testFifoObject { return []testFifoObject{mkFifoObj("foo", 5)} }), }) c := make(chan struct{}) const jobs = 10 for i := 0; i < jobs; i++ { go func() { f.Pop(func(obj interface{}) error { return nil }) c <- struct{}{} }() } runtime.Gosched() f.Close() for i := 0; i < jobs; i++ { select { case <-c: case <-time.After(500 * time.Millisecond): t.Fatalf("timed out waiting for Pop to return after Close") } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: "k8s.io/kubernetes/pkg/kubelet/network/kubenet" "k8s.io/kubernetes/pkg/kubelet/server/streaming" "k8s.io/kubernetes/pkg/kubelet/util/cache" <ins> utilstore "k8s.io/kubernetes/pkg/kubelet/util/store" </ins> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" "k8s.io/kubernetes/pkg/kubelet/dockershim/metrics"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug CheckpointNotFoundError has been completely removed by But GetPodPortMappings method was not updated to return nil, when a checkpoint was not available. What happened: The CNI network (that is done for the shared containers only for windows) call would fail in this case, which is breaking CNI in windows. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Related issue
/sig node /sig windows
/assign
Previous PR for tracking


Question: qs.robinIndex = (qs.robinIndex + 1) % nq queue := qs.queues[qs.robinIndex] if queue.requests.Length() != 0 { <del> currentVirtualFinish := queue.GetVirtualFinish(0, qs.estimatedServiceTime) </del> <ins> currentVirtualFinish := queue.GetNextFinish(qs.estimatedServiceTime) </ins> if currentVirtualFinish < minVirtualFinish { minVirtualFinish = currentVirtualFinish minQueue = queue

I reviewed the current code to see what had been done to generalize request width. I found a few inconsistencies in . A really simple one is at line , where is subtracted from the product but should have been subtracted from just (and their difference multiplied by ). With that fixed, a queue's tracks the expected "virtual time" ( in the original paper) when the oldest waiting request will start. However, computes the expected finish "virtual time" of the J'th request by multiplying by but not . There should be a consistent notion of "virtual time" (AKA ), either always including seats or excluding them. Kubernetes version (use ): latest k8s/apiserver sources Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I suspect it would be better if "virtualTime" is not multiplied by request width. I am working on writing.
/triage accepted
I am running into a dilemma. The whole point of "virtual time" --- R(t) in the original paper --- is that it is shared among all the queues. They all get work done at the same rate, dR/dt. But when we generalize so that requests have different widths (and thus work is the sum over requests of ), even restricting the virtual world so that a queue runs only one request at a time does not save us from the fact that different queues get work done at different rates. Thus, even discarding fairness, and executing one request at a time per queue in the virtual world, it looks like what can be shared among queues is not a matter of work done.
But the current implementation approach makes the assumption that all the non-empty queues get work done at the same rate. As long as we are willing to tolerate the lossages due to that fiction, the queues can share one R(t) meter.
I see both sides of this; I still think it is double counting to use to advance virtual time AND to occupy W seats. OTOH, I also think it is right that we should be fair over the work () -- not just duration. One crazy idea is to add the request to multiple queues, and not execute it until it gets to the head of all the queues it is in.

The number of queues it needs to be to would be (# queues) * W / seats, rounded up I guess.
I think "virtual time" is a fuzzy concept that does not come equipped with guidelines about how to think about it correctly. That is why I followed the style of reasoning in the original paper, it defines a specific function and shows why and how that particular function is useful. In the KEP revision I am preparing, I think the problem statement ()the equations defining intended behavior ()the implementation discussion () make sense. It is in the last of those that is introduced, and I think it is clear that this should be a measure that includes request width.
I don't think it's double counting, because those are different things. And yes - we should be fair over WD - this is exactly why I wanted to push virtual time by WD. - I've seen your KEP PR, just I'm under water. My Fridays are real working time, so I will take a look tomorrow.


Question: return false, bindPathTarget, nil } <ins> func checkSubPathFileEqual(subpath Subpath, bindMountTarget string) (bool, error) { s, err := os.Lstat(subpath.Path) if err != nil { return false, fmt.Errorf("stat %s failed: %s", subpath.Path, err) } t, err := os.Lstat(bindMountTarget) if err != nil { return false, fmt.Errorf("lstat %s failed: %s", bindMountTarget, err) } if !os.SameFile(s, t) { return false, nil } return true, nil } </ins> func getSubpathBindTarget(subpath Subpath) string { // containerName is DNS label, i.e. safe as a directory name. return filepath.Join(subpath.PodDir, containerSubPathDirectoryName, subpath.VolumeName, subpath.ContainerName, strconv.Itoa(subpath.VolumeMountIndex))

When we merged - we introduced comparison of subpath sources: But subpaths source comparison never works properly for all volume types, even if subpath and source remains unchaged. This is because we are comparing wrong things. For a given volume - is usually: But according - the root is - . This means that - if creation of a container times out, then another container can not start until old container exits, because subapth comparison above will be false and kubelet will try to umount the subpath bind mount and it will keep failing with "device is busy" errors. So subpath bind mount can not be reused at all. This is sort of expensive, for containers that use selinux. Because it means each container creation will potentially run recursive chcon again on the entire subpath etc.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig storage
cc
/remove-sig node


Question: return c.convert(sv, dv, s) } <ins> // callCustom calls 'custom' with sv & dv. custom must be a conversion function. func (c *Converter) callCustom(sv, dv, custom reflect.Value, scope *scope) error { if !sv.CanAddr() { sv2 := reflect.New(sv.Type()) sv2.Elem().Set(sv) sv = sv2 } else { sv = sv.Addr() } if !dv.CanAddr() { if !dv.CanSet() { return scope.error("can't addr or set dest.") } dvOrig := dv dv := reflect.New(dvOrig.Type()) defer func() { dvOrig.Set(dv) }() } else { dv = dv.Addr() } args := []reflect.Value{sv, dv, reflect.ValueOf(scope)} ret := custom.Call(args)[0].Interface() // This convolution is necessary because nil interfaces won't convert // to errors. if ret == nil { return nil } return ret.(error) } </ins> // convert recursively copies sv into dv, calling an appropriate conversion function if // one is registered. func (c *Converter) convert(sv, dv reflect.Value, scope *scope) error {

In Go you cannot take the address of a because the map implementation hides that from you. Our conversion library assumes that any reflected map value can have Addr() taken, which causes a panic when you try to do the following for PodInfo conversion: The conversion code tries to call Addr() on the reflect.Value (the map's value, ContainerStatus value object) which fails because it cannot have an address. I worked around it by registering a conversion function on *PodInfo and then manually converting the map.
Oh you mean the reflection code can't addr that? ok sounds like a bug. Non-reflection code can certainly do it:
That playground is taking the address of a value before it's in the struct. You can't take the address of m ----- Original Message -----
Right, because go does magical c++ style references there. But I thought the map copying code handled that case specially already?
I think I see what you're talking about. I'll make a test.
I repro'd. Will fix.
It was actually the source value that caused the trouble. Thanks for the report!
Sorry, should have been clearer. ----- Original Message -----
Note that this map violates our API conventions. It should be a list of named subobjects.
Yeah this field has always been problematic. It's still a bug regardless-- could have triggered copying other maps.


Question: framework.AddOrUpdateLabelOnNode(cs, nodeName, k, v) framework.ExpectNodeHasLabel(cs, nodeName, k, v) defer framework.RemoveLabelOffNode(cs, nodeName, k) // make the nodes have balanced cpu,mem usage <del> createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) </del> <ins> err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) framework.ExpectNoError(err) </ins> By("Trying to launch the pod with podAntiAffinity.") labelPodName := "pod-with-pod-antiaffinity" pod = createPausePod(f, pausePodConfig{

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: numAllNodes: 3, diagnosis: Diagnosis{ PreFilterMsg: "Node(s) failed PreFilter plugin FalsePreFilter", <ins> NodeToStatusMap: NodeToStatusMap{ // They're inserted by the framework. // We don't include them in the reason message because they'd be just duplicates. "node1": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), "node2": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), "node3": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), }, </ins> }, wantReasonMsg: "0/3 nodes are available: Node(s) failed PreFilter plugin FalsePreFilter.", }, { <ins> name:        "nodes failed Prefilter plugin and the preemption also failed", numAllNodes: 3, diagnosis: Diagnosis{ PreFilterMsg: "Node(s) failed PreFilter plugin FalsePreFilter", NodeToStatusMap: NodeToStatusMap{ // They're inserted by the framework. // We don't include them in the reason message because they'd be just duplicates. "node1": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), "node2": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), "node3": NewStatus(Unschedulable, "Node(s) failed PreFilter plugin FalsePreFilter"), }, // PostFilterMsg will be included. PostFilterMsg: "Error running PostFilter plugin FailedPostFilter", }, wantReasonMsg: "0/3 nodes are available: Node(s) failed PreFilter plugin FalsePreFilter. Error running PostFilter plugin FailedPostFilter.", }, { </ins> name:        "nodes failed one Filter plugin with an empty PostFilterMsg", numAllNodes: 3, diagnosis: Diagnosis{

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: - $patch: replace result: mergingList: [] <ins> - description: add new field inside pointers original: mergeItemPtr: - name: 1 patch: mergeItemPtr: - name: 2 result: mergeItemPtr: - name: 1 - name: 2 - description: update nested pointers original: mergeItemPtr: - name: 1 mergeItemPtr: - name: 1 - name: 2 value: 2 - name: 2 patch: mergeItemPtr: - name: 1 mergeItemPtr: - name: 1 value: 1 result: mergeItemPtr: - name: 1 mergeItemPtr: - name: 1 value: 1 - name: 2 value: 2 - name: 2 </ins> sortMergeListTestCases: - description: sort one list of maps original:

PATCH http://IP:8080/api/v1beta3/namespaces/dev--ea6f-49cb-b5bb-/replicationcontrollers/redis-master {"spec":{"replicas":0}} answer: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server responded with the status code 415 but did not return more information", "details": {}, "code": 415 } Before v0.15.0 it had worked well.
That error is returned by go-restful because it did not find any valid route with the same content type as specified in the request: Can you try setting the content-type in your request as is described here: cc
well, I've the same issue which is not fixed by setting the value, at least with the "strategic merge patch". Here's the request: Here's the response body: And here's the log trace of the request in the apiserver pod:
Come from a typo, purposed PR here:
Btw, I'm still having strange issues with sub-objects. Note that it appears only is the sub-object is already set. Here's the request: Here's the API log:
Looks like a bug in strategic merge -- it doesn't deal with pointers correctly:
cc
is it possible to have a milestone for this so I can tell if I should merger ? Thanks.


Question: md5sum "$file" done } <ins> # Install salt from GCS.  See README.md for instructions on how to update these # debs. install-salt() { local salt_mode="$1" if dpkg -s salt-minion &>/dev/null; then echo "== SaltStack already installed, skipping install step ==" return fi echo "== Refreshing package database ==" until apt-get update; do echo "== apt-get update failed, retrying ==" echo sleep 5 done mkdir -p /var/cache/salt-install cd /var/cache/salt-install DEBS=( libzmq3_3.2.3+dfsg-1~bpo70~dst+1_amd64.deb python-zmq_13.1.0-1~bpo70~dst+1_amd64.deb salt-common_2014.1.13+ds-1~bpo70+1_all.deb ) if [[ "${salt_mode}" == "master" ]]; then DEBS+=( salt-master_2014.1.13+ds-1~bpo70+1_all.deb ) fi DEBS+=( salt-minion_2014.1.13+ds-1~bpo70+1_all.deb ) URL_BASE="https://storage.googleapis.com/kubernetes-release/salt" for deb in "${DEBS[@]}"; do if [ ! -e "${deb}" ]; then download-or-bust "${URL_BASE}/${deb}" fi done for deb in "${DEBS[@]}"; do echo "== Installing ${deb}, ignore dependency complaints (will fix later) ==" dpkg --skip-same-version --force-depends -i "${deb}" done # This will install any of the unmet dependencies from above. echo "== Installing unmet dependencies ==" until apt-get install -f -y; do echo "== apt-get install failed, retrying ==" echo sleep 5 done # Log a timestamp echo "== Finished installing Salt ==" } </ins>

I saw this error in on the master after a failed cluster start: I'm looking into this, but I want to create a tracking issue as I think some other people may be hitting this.
It's trying to bring up and failing: The weird thing is that neither that file nor the file have been modified that recently:
Yes, to me it looks like salt just isn't finding the plugin. The last change to _states did add , but I've seen this succeed recently as well (http://kubernetes-). I'm going to try this a few times, to try to figure out whether it is failing every time or not. My guess is that it is failing semi-randomly, so it isn't related to the changes since last-fail.
Aha... this is logged at debug level from I'm guessing it's a salt version mismatch... investigating.
I guess AWS needs to lock salt versions like GCE does:
Confirmed that this is because of upstream salt breakage: - deprecated (?) Not sure what we should do here. I can lock to the same version as GCE, but if we find a bug in that version of salt then we can't update without breaking this script.
We don't write bugs. ;)
Can you just put this in an AWS-specific block? It seems like that should work.
(And by "this" I mean "future Salt variable gunk")
I think I'm going to try to lock AWS to the same version as GCE, and then think more about a long-term plan once AWS is working again!
from the looks of things I think vagrant will have the same problem (?) (Note that I am in the process of developing / testing the branch above, so I don't think you can just copy-and-paste the fix - yet!)
As a short term fix, using 2015.05 (ubuntu trusty salt ppa), was to change from: import as ipaddr to: import as ipaddr this actually seems to be working. might be deprecated. but not in 2015.05. why is there a containerbridge custom state at all btw? why not use network.managed and
- I'm not sure we can just change the import, because I think we have to be compatible with GCE's version of salt also.

I'm not sure of why we did it this way vs other ways of doing things in salt
Does this mean that the official k8s distribution is broken on AWS, BTW? When is the next point release (and where are these discussed)?
We cherry-pick into point releases as necessary, and it's been pretty informal and usually initiated by us (Google). I'd actually be happy to have a discussion about how to initiate it externally (we're going to need a policy for requesting such things). cc
FWIW, policy aside, I'm happy to cut a point release for this once you think is tested well enough.
That would be great, thanks. I'll be testing it pretty hard today, I think. It has occurred to me that other clouds may also be broken. By grepping, it looks like vagrant and vsphere use I am utterly unfamiliar with the vsphere implementation, and I think vagrant sidesteps the issue because on Redhat is skips the call to It might be good to get confirmation that vagrant hasn't fallen in to a different trap though! Let me test it out today (unless people start filing lots of issues, in which case I think we roll it out more urgently...)
I will test out the Vagrant setup today, but I think we should be good because we do not use the same networking setup.
File a bug for a 0.17.1 to get this cherry-picked and assign it to me, as a reminder?
Do you know the status of 17.1? I'm seeing the same problems trying to AWS working, even tried an older release (e.g. v15), and have the same problem. My script looks 32 times on "[salt-master not working yet]" then gives up.
We're cutting one today.
I'm still seeing the same issue as before with the new release, my master starts then times out waiting for salt-master to start. Do you know some spots I can look at to help troubleshoot? Do you want a new issue created to track against?
Please file a new issue and tag thanks. I'm sorry it didn't fix things for you! On May 22, 2015 4:49 AM, "Steve Sloka" wrote:


Question: // RuleWithOperations is a tuple of Operations and Resources. It is recommended to make // sure that all the tuple expansions are valid. type RuleWithOperations struct { <del> // Operations is the operations the admission hook cares about - CREATE, UPDATE, or * // for all operations. </del> <ins> // Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * // for all of those operations and any future admission operations that are added. </ins> // If '*' is present, the length of the slice must be one. // Required. Operations []OperationType `json:"operations,omitempty" protobuf:"bytes,1,rep,name=operations,casttype=OperationType"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Working on other doc bugs, I went looking for what calls admission plugins. I found four call sites for validating plugins, documented in . They correspond to the verbs CONNECT, CREATE, DELETE, and UPDATE. But only two of those verbs are documented at What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Looks like docs are correct in and
The API doc can be updated to call out all four operations and clarify that "all" means "all four admission operations"
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/assign


Question: wantFsUser: nil, }, { <ins> desc: "init and regular containers have runAsUser specified and the same", pod: &v1.Pod{ Spec: v1.PodSpec{ SecurityContext: &v1.PodSecurityContext{}, InitContainers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ RunAsUser: ptr.To[int64](1000), }, }, }, Containers: []v1.Container{ { SecurityContext: &v1.SecurityContext{ RunAsUser: ptr.To[int64](1000), }, }, { SecurityContext: &v1.SecurityContext{ RunAsUser: ptr.To[int64](1000), }, }, }, }, }, wantFsUser: ptr.To[int64](1000), }, { </ins> desc: "all have runAsUser specified and the same", pod: &v1.Pod{ Spec: v1.PodSpec{

projected volumes exclude ephemeral containers in calculating fs user because "SecurityContext is not allowed" : This is no longer technically correct -- ephemeral containers are allowed to specify security context -- but it's a no-op because is guaranteed to be empty at pod creation when fs user is calculated. Still, the comment is wrong so let's clean it up to avoid confusion. /kind cleanup
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /assign
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: } func (jm *ControllerV2) getJobsToBeReconciled(cronJob *batchv1.CronJob) ([]*batchv1.Job, error) { <del> var jobSelector labels.Selector if len(cronJob.Spec.JobTemplate.Labels) == 0 { jobSelector = labels.Everything() } else { jobSelector = labels.Set(cronJob.Spec.JobTemplate.Labels).AsSelector() } jobList, err := jm.jobLister.Jobs(cronJob.Namespace).List(jobSelector) </del> <ins> // list all jobs: there may be jobs with labels that don't match the template anymore, // but that still have a ControllerRef to the given cronjob jobList, err := jm.jobLister.Jobs(cronJob.Namespace).List(labels.Everything()) </ins> if err != nil { return nil, err }

Changing the label on a job template in a CronJob causes any Jobs held by the successful/failed history parameters to stop being tracked, so they never get cleaned up. Jobs with labels that don't match the (upgraded) CronJob, but still have an owner reference, should be subject to the history limit. a CronJob with a job template containing label , and non-zero history limit the CronJob to run and generate some Jobs (default 3 successful, 1 failed) the CronJob with a job template containing label the CronJob to run and generate some more Jobs there are more than 3 successful Jobs/1 failed Job! I believe this is the relevant code: The CronJob controller finds Jobs that it should care about by matching on the labels in the job template, this obviously breaks if the labels in the job template change. 21 Self-hosted No response No response No response No response_
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
Experiencing another side-effect of this bug. If job template labels are updated while a cronjob has status (a scheduled job is running), then that cronjob gets stuck in . The cause seems to be the same as describes: once the job template labels are updated, cronjob controller V2 loses track of the jobs with the old labels, and therefore never updates the status of that active cronjob to indicate that the scheduled job has completed.
/assign


Question: } Expect(err).NotTo(HaveOccurred()) } <ins> func testRCAdoptMatchingOrphans(f *framework.Framework) { name := "pod-adoption" By(fmt.Sprintf("Given a Pod with a 'name' label %s is created", name)) p := f.PodClient().CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, Labels: map[string]string{ "name": name, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  name, Image: nginxImageName, }, }, }, }) By("When a replication controller with a matching selector is created") replicas := int32(1) rcSt := newRC(name, replicas, map[string]string{"name": name}, name, nginxImageName) rcSt.Spec.Selector = map[string]string{"name": name} rc, err := f.ClientSet.Core().ReplicationControllers(f.Namespace.Name).Create(rcSt) Expect(err).NotTo(HaveOccurred()) By("Then the orphan pod is adopted") err = wait.PollImmediate(1*time.Second, 1*time.Minute, func() (bool, error) { p2, err := f.ClientSet.Core().Pods(f.Namespace.Name).Get(p.Name, metav1.GetOptions{}) // The Pod p should either be adopted or deleted by the RC if errors.IsNotFound(err) { return true, nil } Expect(err).NotTo(HaveOccurred()) for _, owner := range p2.OwnerReferences { if *owner.Controller && owner.UID == rc.UID { // pod adopted return true, nil } } // pod still not adopted return false, nil }) Expect(err).NotTo(HaveOccurred()) } func testRCReleaseControlledNotMatching(f *framework.Framework) { name := "pod-release" By("Given a ReplicationController is created") replicas := int32(1) rcSt := newRC(name, replicas, map[string]string{"name": name}, name, nginxImageName) rcSt.Spec.Selector = map[string]string{"name": name} rc, err := f.ClientSet.Core().ReplicationControllers(f.Namespace.Name).Create(rcSt) Expect(err).NotTo(HaveOccurred()) By("When the matched label of one of its pods change") pods, err := framework.PodsCreated(f.ClientSet, f.Namespace.Name, rc.Name, replicas) Expect(err).NotTo(HaveOccurred()) p := pods.Items[0] podClient := f.ClientSet.Core().Pods(f.Namespace.Name) patch := []byte("{"metadata":{"labels":{"name":"not-matching-name"}}}") _, err = podClient.Patch(p.Name, types.StrategicMergePatchType, patch) Expect(err).NotTo(HaveOccurred()) By("Then the pod is released") err = wait.PollImmediate(1*time.Second, 1*time.Minute, func() (bool, error) { p2, err := podClient.Get(p.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) for _, owner := range p2.OwnerReferences { if *owner.Controller && owner.UID == rc.UID { // pod still belonging to the replication controller return false, nil } } // pod already released return true, nil }) Expect(err).NotTo(HaveOccurred()) } </ins>

The merge queue is green with RBAC enabled, and the replicaset controller was missing permission to patch pods until fixed in Pod adoption and release depend on patching pods, which means that there are no merge-queue-blocking tests that test these features. Given that they are complex and rarely tested manually, we need tests that run as part of the merge queue to ensure they function correctly.
I'd like to work on this, where should the tests be ? I see that there are some patch-related tests for the replicaset controller in , and, according to http://submit-, the integration tests are run as part of the merge queue, is that correct? Thanks!
I think we'd want an e2e test that drives behavior externally on a normal cluster (by creating a replicaset, waiting to observe expected pods, modifies replicaset, waits to observe expected results, etc) . many of the integration tests tend to start up fake servers that don't share much in common with real deployments.
Ok thanks, on it.
cc:


Question: // See the flag commentary to understand our assumptions when opening the read-only and read-write ports. <ins> var sem chan bool if s.MaxRequestsInFlight > 0 { sem = make(chan bool, s.MaxRequestsInFlight) } longRunningRE := regexp.MustCompile(s.LongRunningRequestRE) </ins> if roLocation != "" { // Default settings allow 1 read-only request per second, allow up to 20 in a burst before enforcing. rl := util.NewTokenBucketRateLimiter(s.APIRate, s.APIBurst) readOnlyServer := &http.Server{ Addr:           roLocation, <del> Handler:        apiserver.RecoverPanics(apiserver.ReadOnly(apiserver.RateLimit(rl, m.InsecureHandler))), </del> <ins> Handler:        apiserver.MaxInFlightLimit(sem, longRunningRE, apiserver.RecoverPanics(apiserver.ReadOnly(apiserver.RateLimit(rl, m.InsecureHandler)))), </ins> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, MaxHeaderBytes: 1 << 20,

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)