Question: } // create a plugin mgr to load plugins and setup a fake client <del> func newTestWatchPlugin(t *testing.T) (*csiPlugin, *watch.FakeWatcher, string) { </del> <ins> func newTestWatchPlugin(t *testing.T) (*csiPlugin, *watch.RaceFreeFakeWatcher, string) { </ins> tmpDir, err := utiltesting.MkTmpdir("csi-test") if err != nil { t.Fatalf("can't create temp dir: %v", err) } fakeClient := fakeclient.NewSimpleClientset() <del> fakeWatcher := watch.NewFake() </del> <ins> fakeWatcher := watch.NewRaceFreeFake() </ins> fakeClient.Fake.PrependWatchReactor("*", core.DefaultWatchReactor(fakeWatcher, nil)) fakeClient.Fake.WatchReactionChain = fakeClient.Fake.WatchReactionChain[:1] host := volumetest.NewFakeVolumeHost(

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: sc:               resource.sc, pvc:              resource.pvc, pv:               resource.pv, <ins> driverName:       dInfo.Name, </ins> testVolType:      pattern.VolType, nodeName:         dInfo.Config.ClientNodeName, volMode:          pattern.VolMode,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: # managed result of that. Start everything below that directory. log INFO "== Kubernetes addon manager started at $(date -Is) with ADDON_CHECK_INTERVAL_SEC=${ADDON_CHECK_INTERVAL_SEC} ==" <del> # Create the namespace that will be used to host the cluster-level add-ons. start_addon /opt/namespace.yaml 100 10 "" & </del> # Wait for the default service account to be created in the kube-system namespace. token_found="" while [ -z "${token_found}" ]; do

What happened: Add-on manager uses kubectl apply to create the kube-system namespace at start-up. This is redundant because the kube-apiserver creates the namespace in NewBootstrapController. This is a problem because if someone uses apply to make changes to the kube-system namespace, the add-on manager will wipe out these changes. What you expected to happen: Kube-APIServer has the sole responsibility of creating the kube-system namespace. How to reproduce it (as minimally and precisely as possible): kubectl apply a label to the kube-system namespace. restart the add-on manager. Note the labels have been removed. Anything else we need to know?: /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
the contention script wipes out namespace ^^
/cc
i guess it is intentional to have the addon manager to ensure kube-system is not modified, it tries to avoid the case that some cluster admin accidentally delete kube-system that result in a non-functional cluster.
The add-on manager will only perform the check when it restarts. The API Server will also perform this check when it restarts. I think we should only have 1 system responsible for this and I believe that system should be the API server.
trying to have my first contribute here, so is it proper to have the kube-system existence checked before applying the yaml file to create the kube-system namespace once again?
The Kube API Server will check for the existence of the kube-system namespace and create it if absent. It does all of this using client-go requests. The Add-On manager is the only system creating the kube-system by applying (a private) yaml file. I believe the correct fix here is to remove the Add-On managers attempts to create the kube-system namespace entirely. This is especially important if a user writes a CI/CD controller to apply their desired state to the kube-system namespace. I would argue the CI/CD is correct but the add-on manager will randomly wipe out the users changes. The API Server on the other hand will not.
, thanks for your help, so i will remove that line and pull a request later
Trying to get my first PR here. Is the issue still open ?
The issue is still open. You may want to check on though as was trying to fix the issue with that PR.


Question: // KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file // kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config-1.X ConfigMap // Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> // +optional KubeletExtraArgs map[string]string `json:"kubeletExtraArgs,omitempty"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: Limits: v1.ResourceList{ v1.ResourceName("cpu"):           resource.MustParse("10m"), v1.ResourceName("memory"):        resource.MustParse("100Mi"), <del> v1.ResourceName("hugepages-2Mi"): resource.MustParse("50Mi"), </del> <ins> v1.ResourceName("hugepages-2Mi"): resource.MustParse("6Mi"), </ins> }, }, },

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: -tests Since when has it been failing: Unsure Testgrid link: Reason for failure: Unable to pre allocate 50x2MiB huge pages, probably due to memory pressure caused by kernel level caching. If requesting an substantial amount of huge page memory (like the tests), it is recommended to to it as early after boot as possible, or by using the kernel cmd line. Eg. this one: Anything else we need to know: will fix this issue, but in awaiting for that, we can just decrease the page count in the tests.
/sig node /cc derekwaynecarr
I think the reason of railure should be because a single hugepage of x86 is 2MiB as and a single page is 4KiB, not 2KiB.
Yes, is correct. Thanks for catching the typo :smile:
Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. This is probably because the kernel is caching a lot of stuff, making it hard for the kernel to fix it. Ill do some testing with dropping the caches (no user facing stuff will happen, only kernel level) before trying to pre allocate huge pages. /reopen
Reopened this issue. <detailsIn response to : Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: op := &Operation{ ID:       strconv.FormatInt(id, 10), awaiting: from, <del> notify:   make(chan struct{}), </del> <ins> notify:   make(chan bool, 1), </ins> } go op.wait() go ops.insert(op)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue.

<detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.


Question: var totalMillicores int BeforeEach(func() { <del> framework.SkipUnlessProviderIs("gce", "gke") </del> <ins> framework.SkipUnlessProviderIs("gce") </ins> ns = f.Namespace.Name nodes := framework.GetReadySchedulableNodesOrDie(f.Client) nodeCount := len(nodes.Items)

https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
- FYI
Rescheduler is disabled by default so the test won't work. It shouldn't be running in this suite. I'll fix this.
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
This is failing because of Rescheduler is not enabled there. I'm working on this
The test is passing now after redeploying cluster with enabled rescheduler.
This test appears to be broken in the and will be a release blocker. Do you know if there is an issue already tracking this? Can you own this item? cc
Let's reuse this one. I think the problem is that the cluster was not redeployed after was merged. What is the policy of deploying 1.4 cluster? cc
I don't know anything about any policy, so I redeployed 1.4 soak cluster.

https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
It's passing in soak cluster after redeployed it. Thanks! It's still failing in gke-serial.

xref cl/ cc
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
GCE is broken because of DNS migration from RC to Deployment. Reverted in - the test should pass soon. GKE is failing as described in
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}


Question: "k8s.io/kubernetes/pkg/registry/core/service/ipallocator" ) <del> // DefaultServiceIPRange takes a the serviceIPRange flag and returns the defaulted service ip range (if  needed), // api server service IP, and an error func DefaultServiceIPRange(passedServiceClusterIPRange net.IPNet) (net.IPNet, net.IP, error) { </del> <ins> // ServiceIPRange checks if the serviceClusterIPRange flag is nil, raising a warning if so and // setting service ip range to the default value in kubeoptions.DefaultServiceIPCIDR // for now until the default is removed per the deprecation timeline guidelines. // Returns service ip range, api server service IP, and an error func ServiceIPRange(passedServiceClusterIPRange net.IPNet) (net.IPNet, net.IP, error) { </ins> serviceClusterIPRange := passedServiceClusterIPRange if passedServiceClusterIPRange.IP == nil { <del> klog.Infof("Network range for service cluster IPs is unspecified. Defaulting to %v.", kubeoptions.DefaultServiceIPCIDR) </del> <ins> klog.Warningf("No CIDR for service cluster IPs specified. Default value which was %s is deprecated and will be removed in future releases. Please specify it using --service-cluster-ip-range on kube-apiserver.", kubeoptions.DefaultServiceIPCIDR.String()) </ins> serviceClusterIPRange = kubeoptions.DefaultServiceIPCIDR } if size := ipallocator.RangeSize(&serviceClusterIPRange); size < 8 {

API server defaults the service CIDR. We should instead just fail if it is not specified. Defaulting is a wild guess and very likely to be wrong. /sig network /kind bug
For existing clusters that don't currently set it, is setting it to anything other than the current default a breaking change?
Yes. You can not really change it (well, you can expand it, but not totally move it), unless you delete ALL services, including kubernetes.default. I filed this aspirationally. It's not on fire, it's just a dumb thing to have provided a default for.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: able to start with Secret and ConfigMap volumes mounted into the container. release: v1.13 file: test/e2e/storage/empty_dir_wrapper.go <del> - testname: Host path, volume mode default codename: '[sig-storage] HostPath should give a volume the correct mode [LinuxOnly] [NodeConformance] [Conformance]' description: Create a Pod with host volume mounted. The volume mounted MUST be a directory with permissions mode -rwxrwxrwx and that is has the sticky bit (mode flag t) set. This test is marked LinuxOnly since Windows does not support setting the sticky bit (mode flag t). release: v1.9 file: test/e2e/common/host_path.go </del> - testname: Projected Volume, multiple projections codename: '[sig-storage] Projected combined should project all components that make up the projection API [Projection][NodeConformance] [Conformance]'

A long time ago we removed conformance for a similar HostPath test in: At that time a policy was : Current Policy : Note that the conformance tag was in . Which was basically bulk tagging a bunch of tests as conformance tests, So not really specifying a specific reason on why it is a behavior that should be tested. Also note that this is and something that does not work on Windows. And we are expecting the "sticky bit" to be set ... Is this really what we want in a conformance test? So bottom line, does it really need to be a conformance test?
/sig architecture /area conformance
LinuxOnly things should still be in conformance, as I understand it (and hope?!?). Agree with the other reasons.
Yes, you can have LinuxOnly in conformance, that's not the issue here. The issue is that it does violate the current policy, and looks like it was to conformance prior to that but probably should have been removed. So it seems reasonable to me.


Question: select { case err := <-wc.errChan: <del> if err == context.Canceled { </del> <ins> if isCancelError(err) { </ins> break } errResult := transformErrorToEvent(err)

TestChangeCRD " TestChangeCRD": 37, Recently, not sure the date. No response_ /sig api-machinery
also flaking in weirdly, this doesn't show up at all in
cc I can't tell if this is related to the openapi change or not
Actually, I think this is an issue either with the test or with watch behavior, not sure which. The test starts 10 watches, lets them run, then waits for them to exit cleanly, and instead, they are getting watch errors. I can't tell if the watch errors are expected or not, or if server or watch client behavior changed here or not. Opened to debug further
and stand out as a recent change to watching
cc on whether the watching changes in could explain this integration test getting watch errors it didn't previously get
also stands out
would explain this integration test getting watch errors it didn't previously get?
the filter is enabled only when the server run option has a positive time duration (the default is zero). I searched the k/k repo, only the unit tests in enables this filter: I did a quick scan of the logs, did not see any graceful termination activity either. I will take a deeper look at the logs
and all merged changes were limited to that filter?
This is a Go test, so the entire test output is treated as the failure message. That output is very different from one failure to the other, so in triage we probably have x different "clusters" with one failure in each, instead of one cluster with x failures. I don't know what triage does with such small clusters.
we have changes in only the the filter adds to the request context, I just searched the code base, did not find any other call site for by default, should block (on a nil channel) If the test is receiving an event, it implies that the following case is executing, right?
xref: ??
Thank you all for deflaking.
/triage accepted


Question: message RemoveImageResponse {} message NetworkConfig { <del> // CIDR to use for pod IP addresses. </del> <ins> // CIDR to use for pod IP addresses. If the CIDR is empty, runtimes // should omit it. </ins> string pod_cidr = 1; }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In my test setup I allocate POD CIDRs manually, bridge is configured before starts. Important settings: on example node: on master: Then I see confusing entries in logs (): Fortunately everything keeps working, thanks to this condition: , i.e. it is not propagated to CNI plugin. It appears that this very line tries to set the CIDR to whatever the master returns during node status update: What you expected to happen: / use the pod CIDR given at command line and never mess with it Actually I'm a bit confused if manual per-node pod CIDR assignment is supported at all, and if I'm doing it correctly. I'm aware it's unusual thing to do in production environment, yet for testing it seems perfectly legitimate. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu 17.10 Kernel (e.g. ): Install tools:
/sig network
Kubelet need to make a check of CIDR before
/assign


Question: }) } <ins> func TestDropInvalidRule(t *testing.T) { for _, testcase := range []bool{false, true} { t.Run(fmt.Sprintf("tcpLiberal %t", testcase), func(t *testing.T) { ipt := iptablestest.NewFake() fp := NewFakeProxier(ipt) fp.conntrackTCPLiberal = testcase fp.syncProxyRules() expected := dedent.Dedent(` *filter :KUBE-NODEPORTS - [0:0] :KUBE-SERVICES - [0:0] :KUBE-EXTERNAL-SERVICES - [0:0] :KUBE-FIREWALL - [0:0] :KUBE-FORWARD - [0:0] :KUBE-PROXY-FIREWALL - [0:0] -A KUBE-FIREWALL -m comment --comment "block incoming localnet connections" -d 127.0.0.0/8 ! -s 127.0.0.0/8 -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP`) if !testcase { expected += "n-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP" } expected += dedent.Dedent(` -A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT -A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT COMMIT *nat :KUBE-NODEPORTS - [0:0] :KUBE-SERVICES - [0:0] :KUBE-MARK-MASQ - [0:0] :KUBE-POSTROUTING - [0:0] -A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS -A KUBE-MARK-MASQ -j MARK --or-mark 0x4000 -A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN -A KUBE-POSTROUTING -j MARK --xor-mark 0x4000 -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE COMMIT `) assertIPTablesRulesEqual(t, getLine(), true, expected, fp.iptablesData.String()) }) } } </ins> func TestMasqueradeRule(t *testing.T) { for _, testcase := range []bool{false, true} { ipt := iptablestest.NewFake().SetHasRandomFully(testcase)

When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. However, there is no corresponding session link on the host, and the host sends a reset packet, causing the session link to be interrupted connection not reset This problem can be solved by command iptables -t filter -I INPUT -p tcp -m conntrack --ctstate INVALID -j DROP Similar But this issue "drop" is placed on the forward chain, our scenario needs to be placed on the input chain. $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} <details</details<details</details<details</details<details</details<details</details
/sig network is this about pods with hostNetwork: true ? if not, why is this a kube-proxy issue if the problem is in INPUT?
/assign
Yeah, to be clear - filter INPUT applies to packets which are going to be delivered to a process. Is that kube-proxy's job to define?
The "How can we reproduce" sounds a bit like this always happens. Here is a better instruction
IMO is the best description of the problem:
This may be a better solution than let add rules in the INPUT chain
hehe, ironically we should decide to or , one is a sysctl call the other is one (or two) iptables rules. The iptables rule has also the side effect of breaking people (cc ) If we switch to set the option, the change does not seem to be backportable. I tend to agree with Lars, remove the iptables INVALID drop rule and set the sysctl, we are already setting more host sysctls what do you think?

it will be nice to have some iptables hackers opinion too
I tried to recreate this problem on virtual en (kvm/qemu) by requsting a 100MB file with curl and do; to provoke some packet loss. The transfer took longer time, but never failed.
IMO kube-proxy should not set any sysctls that are not literally required for functionality that the user has explicitly opted into (eg for most network plugins). Kube-proxy does not own the host network namespace, and it should not be doing things that will affect other people's host-network traffic, because if we do it's going to break some users. (See also: .) We can document that we think it's a good idea for users/distros to set , and if we think it's almost always a better idea than not, we can warn at startup if it's not set, but (IMO) we shouldn't set it ourselves. If we can get the same effect with an iptables rule that only affects kube-proxy's own traffic, then I think that's better than setting a sysctl that will also affect non-kube-proxy traffic. If I understand the situation here correctly, if kube-proxy a drop rule for the invalid conntrack packets, but then the administrator set , then the result would be that conntrack would not mark some packets as invalid, and so our drop rule would just not get hit, and so our drop rule wouldn't interfere with the "better" sysctl-based solution?
Agree in general. We have done it in the past (e.g. route-localnet) and mostly it seems like a not-great idea. Also agree.
If we see this as a known bug, please consider documenting it in - even if latest Kubernetes includes a fix (we can nevertheless point that out!)
I'll fix this in the next few days.
From further googling, it seems like there have been situations in the past where people were setting that sysctl to work around bugs in conntrack, which were later fixed. I think this may be another such situation; it seems like our problem is that there is some state involving dropped or retransmitted packets which linux would cope with fine in the non-conntrack case, but which it doesn't handle in the conntrack case, unless "be_liberal" is set, and that seems like a bug.

Every time I look away from this set of issues/PRs for longer than 15 minutes I forget the exact scenario that causes the bug, but if someone could summarize it very clearly we could try dragging some kernel developers in. (The regression test in the e2e suite creates a packet with an intentionally-bad sequence number, but it's not clear to me what sort of real-world issue that's supposed to be representing.)
Thanks now that was an excellent article! Well written (not TL;DR that is), and IMO perfect technical level. If there is some collection of recommended reading for persons who want to know about K8s networking, this article should be in it.
Hi thank you for the , it's really helpful! I have some questions: , should we update this sentence accordingly ==we use the above workaround, would there any side effect for the cluster network or cluster nodes?
That says "this problem will occur if conntrack is unable to recognize a packet", but it doesn't explain the real-world scenario that causes conntrack to be unable to recognize a packet. If we go to the kernel developers and say "manually injecting random incorrect packets into a conntrack'ed connection causes it to get confused", then they'll likely respond either "don't do that then" or "use ". But if we can tell them something like "if a connection has a duplicated packet after a dropped packet, then conntrack gets confused", then that sounds a lot more like a bug in conntrack that they ought to fix, such that then everything will just work for everyone in the future without needing any DROP rules or sysctls.
So do we need to revert the drop rules? As mentioned in , it now seems that this makes sense.
There are three potential "fixes" here: this is unambiguously a conntrack bug, then we should figure out the details, and get the kernel devs to fix it. However, getting kernel bug fixes into all k8s clusters in the world takes "a long time", so even if it is a kernel bug, we should still think about the other fixes. cluster admins can use . We should advertise this better, but we feel that it would be dubious to have kube-proxy set this flag itself ().

per we could probably tweak the existing rule so that it didn't interfere with non-k8s packets ~and as suggested by the OP of this issue, we could add a similar rule to the INPUT chain (though this would need a bit of further thinking about to make sure we weren't introducing any new conflicts with non-k8s traffic)~.
We could revert the DROP rules and tell people to use the sysctl instead (and actually, probably that's a better answer for this issue than adding a new DROP rule to the INPUT chain is). But we'd need to warn users in advance if we were going to do that, since it would break existing clusters.
Totally agree, This is a better fix for this issue. If this is a kernel bug and it gets fixed someday, we can easily tell the user that you don't need to set anymore. Also, If we revert the DROP rule, which can solve the routing asymmetry issue mentioned in , which is a big fix for us Yes, We need to document it and tell the user. and In order not to break the existing cluster, Do we need a flag to decide if we need the DROP rule? I'll make it default to , and make it consistent with the current logic.
but the change to only apply the DROP rule to pod traffic should also fix that We should not require administrators to choose between "high-traffic TCP services work" and "asymmetric routing works". The current DROP rule is broken. We should either fix it to work with asymmetric routing, or else remove it and tell people who needed it to use the sysctl instead. In the latter case, we might want to make the transition via a feature gate. Then people who need asymmetric routing to work can enable the feature gate in Alpha (to make kube-proxy stop creating the DROP rule), and people who need high-traffic services to work but who didn't get the memo about using the sysctl can disable the feature gate in Beta until they have a chance to update their node configs to set the right sysctl.
Refer to , The DROP rule to pod doesn't fix our case. In our case, the pods of the external cluster subnet failed to access the pods of the cluster subnet to connect due to asymmetric routing, So the packet still matches The DROP rule to pod. Do we need a KEP for this or only a feature gate?

In general, feature gates imply KEPs (and any change that will take multiple releases to complete, where we want to make sure we don't forget about it halfway through, should get a KEP, because we have processes for tracking those). It doesn't have to be a big KEP, and some of the sections of the KEP template will be irrelevant and can be omitted, because you're not introducing a new feature. (See for example.)
We are going to break users, unfortunatly features gates does not have the effect that we are assuming, and we know that all the users have to set the "ipconntracktcpbeliberal" because eventually some connections will be broken. I do no like to set sysctl, but 'm not in favor of replacing the rule and leaving the problem to users that may not even know what a sysctl is ... agree the DROP rule is broken and breaks some esoteric but legit scenarios, but we are already setting sysctls, so I think that if we don't find any alternative to the DROP rule we should try to set the sysctl for the user by default
If we skip a KEP, make sure that the release team know to track the details (comms, docs, etc) as if it were a KEP. This is an important thing to track.
Should we at least triage this as Accepted?
/triage accepted Some alternatives DROP rule and ask users to set sysctl (I'm not in favor per my previous comment and the least surprise principle) DROP rule and we set sysctl set DROP rule if is not set the DROP rule with a flag/config so users can opt-out if don't want to set it (they can decide to use the sysctl or live with that risk) ...
I can accept KEP, but I think this may take multiple releases to complete. how about this? We remove the DROP rule Control the sysctl with a flag or a feature-gate. If it's a flag, I believe it's relatively simple and we can finish it in 1.29. In the case of feature-gate, this may take multiple releases to complete.
If sets the to true by default, we can safely remove the DROP rule
Yes, but if we were doing it that way, that would be because we considered that to be a feature rather than a bug.

We want to avoid breaking working clusters on upgrade. Maybe just having a release note saying "you may need to pass to kube-proxy" is enough. I think we agreed we don't want to change any sysctls except by explicit admin request (because that affects the behavior of non-kubernetes networking as well).
Agree, but we need to think in the users, this is a problem where the fix that solve a sublte and complex bug that may affect 100% of kubernetes users and most probably they don't know they are affected, impact a esoteric network configuration on the cluster networking, and the solution we are offering is "please read the release notes", that we know only 20% of users do ... I think that advanced users can opt-out to the DROP rule if we don't want to mess with the sysctl,
If setting was a no-brainer good idea, then the kernel would just have that behavior by default. There must be some reason why they don't, and so I worry that if we set that flag by default, then we're just going to start breaking some other subset of users. Also, if it turns out that this is all just a conntrack bug, which eventually gets fixed, then we wouldn't want to be setting the sysctl after it got fixed. But at that point it would be likely that there were some users unknowingly relying on the sysctl for some other side effect.
We could potentially have an iptables rule to count invalid conntrack packets, and expose the count as a metric
I completely and absolutely agree. we should not make that sysctl a default in kube-proxy ... what if we read the sysctl and install the DROP rule only if it is not set? Users that apply the sysctl we no longer have the DROP rule
/assign let me send a PR , I think that approach covers all cases
FWIW, my kernel conntrack informant tells me that there really aren't any bad side effects of setting it ("I don't see anything that could break by enabling this mode"), and in fact OVN sets it unconditionally on hosts it's running on. He also tells me that the bug (at least in ) is fixed in kernel ~6.2~ 6.1 ("").

So I'm not sure if that means "we should just go ahead and set the sysctl because it's harmless" or "we should just keep it like it is now and not set the sysctl since it won't be needed in the future"... (NB: if the latter we still need a docs update to go along with this PR, telling admins about the sysctl.)
If we can determine that the kernel fixes the issue in a certain version, then I think we can do this: Do not install the DROP rule.If the kernel version of the host is greater than the safe kernel version, we do not set sysctl.If less than, we have a flag to set sysctl. I'll make it default to 1. (refer to ), And let the user know that we may remove this sysctl in the future.
Exactly my thought. I made this mistake with route_localnet, and we still pay for it. Let me do some internal outreach too, see if I can get a concurring opinion?
Kernel ppl here concur: "I say : if kernel can not be fixed, set tcpbeliberal to one " "having Kubernetes set net.netfilter.nfconntracktcpbeliberal=1 always SGTM, since you want it to work on any kernel version" argues for a flag, default to true (or a hegative flag, defsult false) which seems a bit paranoid, but probably smart :) Are we all in agreement? Who wants to do the PR?
already has a PR to add to the existing set of configurable conntrack sysctls in kube-proxy () so it's just a matter of flipping the default
If the sysctl defaults to true, I think this still keeps backward compatibility, So we don't need the DROP rule anymore.
I just realized that we need this to be a flag because there are cases that run kube-proxy on environments with readonly /sys subsystem, so trying to set it always will fail will and make kube-proxy unusable for those environments
if people does not set it, they will hit the bug until they use kernel 6.1
I mean this sysctl defaults to true in kube-proxy, I can't imagine why users would need to set it to false unless the kernel version is safe (6.1). Of course, we can also document that the user that it is better not to do this.

There are cases where we CAN'T set sysctls but should not hard-fail kube-proxy. On Wed, Sep 6, 2023, 11:25 PM Cyclinder
after setting the jobs for nftables and hitting the conntrack bug continuously I think we should default the flag to true and set the sysctl nfconntracktcpbeliberal to 1 , We are already setting some sysctl and for rootless or other environment they can always opt out, this is what we do in kind
Sounds OK to me. Risk should be ~zero, right?
Yes, I assumed the risk is when to 1, we do not set it by default, so default to 0(see ). I'd be happy with it, Can I have this one?
Talked with offline, I don't have very strong arguments to make this enable by default , so let's wait for more user feedback, nftables will move to beta this release, we can revisit if we have more feedback from users
we are considering turning on by default in our kube-proxy 1.29+ deployments. Based on the recent discussion here that seems fine and the risk should "be ~zero"(), correct? And you are all even discussing turning this on by default in kube-proxy in some future version? One thing I'm a little confused on: is doing the same thing as turning on ? So by setting you're just turning on behavior that will be on by default in 6.1. Did I get that right?
Not exactly. affects a handful of situations. 6.1 only "normalized" one of them. The general intent of is something like "make conntrack work better with packets from broken TCP implementations". But the netfilter devs realized that one of the places it was being used was actually something that could happen due to totally normal TCP packet drops/retransmissions. And so in 6.1+, that particular situation is just handled correctly automatically, whether is enabled or not. My argument against this is that once we do it, we have to keep doing it forever, because people might start unwittingly depending on the other side effects of that sysctl. And particularly given that people never hit the bug anyway.

(It seems like we got more reports of the "my cluster is broken because of the DROP rule" bug than we ever got of the "my cluster is broken because of conntrack invalid packet handling" bug.) day once everyone has moved to 6.1 the bug will be gone forever ISTM that it's better to still not set it by default.
Thanks for the clarification on the 6.1 bugfix That's true and they probably will. I'm just trying to leverage some of the earlier discussion here to make a decision about the safety of flipping this to true, as it seems like the concern was not too high / basically nil.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper"

<img width="1064" alt="image" src=""### Failure cluster /kind failing-test


Question: "set_image_test.go", "set_resources_test.go", "set_selector_test.go", <ins> "set_serviceaccount_test.go", </ins> "set_subject_test.go", "set_test.go", ], data = [ "//examples:config", <ins> "//test/fixtures", </ins> ], library = ":go_default_library", deps = [ "//pkg/api:go_default_library", <ins> "//pkg/api/testapi:go_default_library", "//pkg/apis/apps:go_default_library", </ins> "//pkg/apis/batch:go_default_library", "//pkg/apis/extensions:go_default_library", "//pkg/apis/rbac:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: logger.V(10).Info("filtering based on resource claims of the pod", "pod", klog.KObj(pod), "node", klog.KObj(node), "resourceclaim", klog.KObj(claim)) switch { case claim.Status.Allocation != nil: <del> if nodeSelector := state.availableOnNodes[index]; nodeSelector != nil { </del> <ins> if nodeSelector := state.informationsForClaim[index].availableOnNode; nodeSelector != nil { </ins> if !nodeSelector.Match(node) { logger.V(5).Info("AvailableOnNodes does not match", "pod", klog.KObj(pod), "node", klog.KObj(node), "resourceclaim", klog.KObj(claim)) unavailableClaims = append(unavailableClaims, index)

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: } tests := []struct { <del> name         string pendingPod   *v1.Pod addedPod     *v1.Pod existingPods []*v1.Pod nodes        []*v1.Node services     []*v1.Service </del> <ins> name                 string pendingPod           *v1.Pod addedPod             *v1.Pod existingPods         []*v1.Pod nodes                []*v1.Node services             []*v1.Service expectedAntiAffinity topologyToMatchedTermCount expectedAffinity     topologyToMatchedTermCount </ins> }{ { name: "no affinity exist",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Maybe it should be What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Scheduling
I feel it is a valid error. But need confirmation from sig-scheduling maintainers. ... is it okay if I work on this?
ok
Yeah, this is a bug in preemption
/assign


Question: Version *string `protobuf:"bytes,1,opt,name=version" json:"version,omitempty"` // The name of the container runtime. RuntimeName *string `protobuf:"bytes,2,opt,name=runtime_name,json=runtimeName" json:"runtime_name,omitempty"` <del> // The version of the container runtime. </del> <ins> // The version of the container runtime. The string should be // semver-compatible. </ins> RuntimeVersion *string `protobuf:"bytes,3,opt,name=runtime_version,json=runtimeVersion" json:"runtime_version,omitempty"` <del> // The API version of the container runtime. </del> <ins> // The API version of the container runtime. The string should be // semver-compatible. </ins> RuntimeApiVersion *string `protobuf:"bytes,4,opt,name=runtime_api_version,json=runtimeApiVersion" json:"runtime_api_version,omitempty"` XXX_unrecognized  []byte  `json:"-"` }

umbrella issue: kuberuntime uses to ]parse the version string of the api version]() from the runtime. When using with docker, it complains that Maybe we should just enforce that the api version returned by the runtime should be semver compatible. /cc
This is mentioned in . :) For now, we can add a after the docker apiversion to make it semver compatible.
Thanks for the reminder! We should make it clear in the API. This blocks kubelet from starting up, so we should fix it first.
I've checked the code, currently is only used in when checking the docker version compatibility. It is not exposed to the user by any means, so it should be fine to make some change ourselves, such as adding a . However, if we want to expose this to the user via in the future, it may make user confusing.


Question: UseEndpointSlices      bool OOMScoreAdj            *int32 ConfigSyncPeriod       time.Duration <del> HealthzServer          *healthcheck.ProxierHealthServer </del> <ins> HealthzServer          healthcheck.ProxierHealthUpdater </ins> } // createClients creates a kube client and an event client from the given config and masterOverride.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.

/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved


Question: // HPAUpgradeTest tests that HPA rescales target resource correctly before and after a cluster upgrade. type HPAUpgradeTest struct { <del> rc *common.ResourceConsumer </del> <ins> rc  *common.ResourceConsumer hpa *autoscalingv1.HorizontalPodAutoscaler </ins> } func (HPAUpgradeTest) Name() string { return "hpa-upgrade" }

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "testing" api_v1 "k8s.io/kubernetes/pkg/api/v1" "github.com/stretchr/testify/assert" ) func TestGetClusterName(t *testing.T) { // There is a single service ns1/s1 in cluster mycluster. service := api_v1.Service{ ObjectMeta: api_v1.ObjectMeta{ Namespace: "ns1", Name:      "s1", Annotations: map[string]string{ ClusterNameAnnotation: "mycluster", }, }, } name, err := GetClusterName(&service) assert.NoError(t, err) assert.Equal(t, "mycluster", name) } func TestSetClusterName(t *testing.T) { // There is a single service ns1/s1 in cluster mycluster. service := api_v1.Service{ ObjectMeta: api_v1.ObjectMeta{ Namespace: "ns1", Name:      "s1", }, } err := SetClusterName(&service, "mytestname") assert.NoError(t, err) clusterName := service.Annotations[ClusterNameAnnotation] assert.Equal(t, "mytestname", clusterName) } </ins>

Federated controllers will oftentimes have to crreate/update/delete the same/similar structures in multiple clusters. Given that the roundtrip time to remote clusters can be high (like, lets say, 1 sec) these operations should be done in parallel so that the main controller goroutine is not blocked for long. Part of


Question: featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/kubernetes/pkg/controller/testutil" "k8s.io/kubernetes/pkg/features" <del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </del> ) var timeForControllerToProgressForSanityCheck = 20 * time.Millisecond

Events from node taint manager doesn't report involved object apiVersion: ! apiVersion is reported as other events. For example: ! a pod on node A (not master) off node A is reported, but without apiVersion in involvedObject. The apiVersion is not reported in The issue is still not fixed in master: ! <details</details<detailsIrrelevant </details<details</details<details</details<details</details<details</details
/sig node /sig apps
/assign In the latest version, the issue is confirmed
one suggestion you should wait for the corresponding SIG to triage the issue and accept it before sending PRs, the chances of your PR getting reviewed and approver are much higher
You are right sorry about that. It won’t happen again.
no no worries, it was a suggestion, is not that you did something wrong, but it is possible that your PR doesn't get reviews and you get frustrated
/remove-sig node please re-add if this is node-related issue
I think it is a node related issue. The root cause in taint manager.
/sig node
/assign /triage accepted


Question: return "", fmt.Errorf("error tagging volume %s: %q", volumeName, err) } <ins> // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if len(volumeOptions.KmsKeyId) > 0 { err := c.waitUntilVolumeAvailable(volumeName) if err != nil { if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") } return "", err } } </ins> return volumeName, nil } <del> // checkEncryptionKey tests that given encryption key exists. func (c *Cloud) checkEncryptionKey(keyId string) (missing bool, err error) { input := &kms.DescribeKeyInput{ KeyId: aws.String(keyId), } _, err = c.kms.DescribeKey(input) if err == nil { return false, nil </del> <ins> func (c *Cloud) waitUntilVolumeAvailable(volumeName KubernetesVolumeID) error { disk, err := newAWSDisk(c, volumeName) if err != nil { // Unreachable code return err </ins> } <del> if awsError, ok := err.(awserr.Error); ok { if awsError.Code() == "NotFoundException" { return true, fmt.Errorf("KMS key %s not found: %q", keyId, err) </del> <ins> err = wait.Poll(encryptedCheckInterval, encryptedCheckTimeout, func() (done bool, err error) { vol, err := disk.describeVolume() if err != nil { return true, err </ins> } <del> } return false, fmt.Errorf("Error checking KSM key %s: %q", keyId, err) </del> <ins> if vol.State != nil { switch *vol.State { case "available": // The volume is Available, it won't be deleted now. return true, nil case "creating": return false, nil default: return true, fmt.Errorf("unexpected State of newly created AWS EBS volume %s: %q", volumeName, *vol.State) } } return false, nil }) return err </ins> } // DeleteDisk implements Volumes.DeleteDisk

This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events.

Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV.

And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available".

I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:


Question: return data, utilerrors.NewAggregate(errlist) } <del> // sortLines sorts the strings generated from map based data // (annotations and labels) func sortLines(values string) string { splitted := strings.Split(values, "n") sort.Strings(splitted) return strings.Join(splitted, "n") } </del> func (d *downwardAPIVolume) GetPath() string { return d.plugin.host.GetPodVolumeDir(d.podUID, utilstrings.EscapeQualifiedNameForDisk(downwardAPIPluginName), d.volName) }

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: "k8s.io/kubernetes/pkg/util/rlimit" "k8s.io/kubernetes/pkg/version" "k8s.io/kubernetes/pkg/version/verflag" <del> nsutil "k8s.io/kubernetes/pkg/volume/util/nsenter" </del> "k8s.io/kubernetes/pkg/volume/util/subpath" "k8s.io/utils/exec" <del> "k8s.io/utils/nsenter" </del> ) const (

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "github.com/spf13/cobra" <del> appsv1 "k8s.io/api/apps/v1" </del> batchv1 "k8s.io/api/batch/v1" batchv1beta1 "k8s.io/api/batch/v1beta1" corev1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: "scheme": "HTTPS", "host": "127.0.0.1", "port": {{secure_port}}, <del> "path": "/healthz?exclude=etcd&exclude=kms-provider-0&exclude=kms-provider-1" </del> <ins> "path": "/livez?exclude=etcd&exclude=kms-provider-0&exclude=kms-provider-1" </ins> }, "initialDelaySeconds": {{liveness_probe_initial_delay}}, "timeoutSeconds": 15

Add readyz to GCE manifest for readiness checking on the apiserver. /assign
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "//pkg/kubelet/apis/resourcemetrics/v1alpha1:go_default_library", "//pkg/kubelet/container:go_default_library", "//pkg/kubelet/prober:go_default_library", <ins> "//pkg/kubelet/server/metrics:go_default_library", </ins> "//pkg/kubelet/server/portforward:go_default_library", "//pkg/kubelet/server/remotecommand:go_default_library", "//pkg/kubelet/server/stats:go_default_library",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: func() { service.mu.Lock() defer service.mu.Unlock() <ins> // If the service is nil, that means it hasn't yet been successfully dealt // with by the load balancer reconciler. We can trust the load balancer // reconciler to ensure the service's load balancer is created to target // the correct nodes. if service.service == nil { return } </ins> if err := s.lockedUpdateLoadBalancerHosts(service.service, hosts); err != nil { glog.Errorf("External error while updating TCP load balancer: %v.", err) servicesToRetry = append(servicesToRetry, service)

In , I found a logic error (when I had a broken cloud implementation): If createLoadBalancerIfNeeded returns err, then cachedService will have service = nil. This causes updateLoadBalancerHosts to panic. It would be easy to skip cacheService if service=nil, but I wasn't sure whether this should then be included in servicesToRetry or not. Another problem (by inspection) is that I think there's a race condition where updateLoadBalancerHosts can run immediately after getOrCreate, before the service field has been set.
/cc
Great catches, Justin, thanks! I believe that simply not processing cached services with a nil api.Service object in the node pool reconciler is the correct approach. A nil Service object implies that the load balancer reconciler still intends to process it, and the LB reconciler always uses the most up-to-date list of nodes. I'll send a PR ASAP.


Question: switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err): </del> <ins> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details