Question: hdlr := NewExampleHandler(supportedVersions, false /* permitDeprecatedDir */) hdlr.AddPluginName(pluginName) <ins> c := make(chan struct{}) go func() { defer close(c) require.True(t, waitForEvent(t, exampleEventValidate, hdlr.EventChan(p.pluginName))) require.False(t, waitForPluginRegistrationStatus(t, p.registrationStatus)) }() </ins> w := newWatcherWithHandler(t, hdlr, false /* testDeprecatedDir */) defer func() { require.NoError(t, w.Stop()) }() <del> require.True(t, waitForEvent(t, exampleEventValidate, hdlr.EventChan(p.pluginName))) require.False(t, waitForPluginRegistrationStatus(t, p.registrationStatus)) </del> <ins> select { case <-c: return case <-time.After(wait.ForeverTestTimeout): t.Fatalf("Timeout while waiting for the plugin registration status") } </ins> } func waitForPluginRegistrationStatus(t *testing.T, statusChan chan registerapi.RegistrationStatus) bool {

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: &createPodSetsOp{}, &churnOp{}, &barrierOp{}, <del> // TODO(#93793): add a sleep timer op to simulate waiting? </del> <ins> &sleepOp{}, </ins> // TODO(#94601): add a delete nodes op to simulate scaling behaviour? } var firstError error

Note: this issue should be viewed in the context of . What would you like to be : Currently we support three types of ops: , , . This issue proposes adding a op that would simulate user action by causing the main thread to sleep. Note: I don't have a use-case for this in mind, but one can come up with trivial and contrived examples easily. Why is this needed: Not clear at this time. This is just an open discussion where others can chime in with whether this will be useful. /sig scheduling /sig testing (Not assigning myself until we can figure out if/how we want this.)
when you get a chance, can you group all enhancements issues relate to the new benchmark framework under one issue with a task list.
/assign
see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/unassign
I'd like to take this one /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is still an active pull request for this issue. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : There is still an active pull request for this issue. /reopen </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apiserver/pkg/admission" <ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" </ins> "k8s.io/apiserver/pkg/features" "k8s.io/apiserver/pkg/storage/names" utilfeature "k8s.io/apiserver/pkg/util/feature"

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: "--concurrent-service-endpoint-syncs=10", "--concurrent-gc-syncs=30", "--concurrent-namespace-syncs=20", <ins> "--concurrent-job-syncs=10", </ins> "--concurrent-replicaset-syncs=10", "--concurrent-resource-quota-syncs=10", "--concurrent-service-syncs=2",

What would you like to be : The ability to set for the kube-controller-manager at runtime. It looks like the option exists in the code, but a mechanism is not available to set it via kops. If I understand the code correctly, a flag needs to be in order to be able to set the value using kops. Why is this needed: We want to shift a lot of our batch processing jobs from running directly on our CI servers to running as jobs on Kubernetes. Unfortunately if too many jobs are launched on kubernetes, they begin to back up. We've noticed up to a 10 minute delay between when the logs indicate that a job has finished and when the kubernetes API actually reflects the same on our small ~30 node cluster.
/sig scalability
I think scalability is the best fit since this is impacting our ability to fully utilize and scale how we use our kubernetes cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Is there anything I can do to help get landed?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-sig api-machinery /sig apps
/triage accepted /remove-lifecycle rotten
can you please describe your use case in a more detail? how How this affects you, and how this flag would help to achieve your goals?
-- I left a comment describing my use case in the PR:


Question: mu.Lock() for i := range *prioritizedList { host, score := (*prioritizedList)[i].Host, (*prioritizedList)[i].Score <ins> if glog.V(10) { glog.Infof("%v -> %v: %v, Score: (%d)", util.GetPodFullName(pod), host, ext.Name(), score) } </ins> combinedScores[host] += score * weight } mu.Unlock()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In pkg/scheduler/core/generic_scheduler# , add some debug info: each scheduler extender's score and its name for each pod, e.g. And we need add a function(e.g. ) in interface to get extender's name or similar identity. For , it might be enough to return its url . Why is this needed: If there are two or more scheduler extenders configured for scheduler(and we have now), it will be very useful to debug why kubelet nodes get so high/low score. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Why don't you send a PR instead?
Thanks. PR is sent. PTAL.


Question: return false, false, fmt.Errorf("context cancelled") case e := <-w.Events: switch e.Op { <del> case fsnotify.Write: </del> <ins> case fsnotify.Write, fsnotify.Rename, fsnotify.Remove, fsnotify.Chmod: </ins> return true, false, nil case fsnotify.Create: <del> fallthrough case fsnotify.Rename: fallthrough case fsnotify.Remove: fallthrough case fsnotify.Chmod: return true, true, nil </del> <ins> return true, filepath.Base(e.Name) == logName, nil </ins> default: klog.ErrorS(nil, "Received unexpected fsnotify event, retrying", "event", e) }

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: // WalkFunc which is called for each file and directory in the pod cgroup dir visitor := func(path string, info os.FileInfo, err error) error { <ins> if err != nil { glog.V(4).Infof("cgroup manager encountered error scanning cgroup path %q: %v", path, err) return filepath.SkipDir } </ins> if !info.IsDir() { return nil } pids, err = getCgroupProcs(path) if err != nil { <del> glog.V(5).Infof("cgroup manager encountered error getting procs for cgroup path %v", path) </del> <ins> glog.V(4).Infof("cgroup manager encountered error getting procs for cgroup path %q: %v", path, err) </ins> return filepath.SkipDir } pidsToKill.Insert(pids...)

https://k8s- Failed: [] Downward API should provide default from node allocatable [Conformance] {E2eNode Suite}
this looks like the same issue fixed
For reference, this showed the following panic: which is fixed in referenced pr.


Question: fuzz "github.com/google/gofuzz" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" "k8s.io/kubernetes/pkg/apis/batch" <ins> api "k8s.io/kubernetes/pkg/apis/core" </ins> "k8s.io/utils/pointer" )

pull-kubernetes-unit: TestRoundTripTypes Unknown No response No response /sig api-machinery
/kind bug /triage accepted
cc
cc I think fuzzing is missing a default for the newly type, so when fuzzing adds an entry, the original doesn't match the round-tripped data (which applies the API default)
/priority important-soon /remove-sig api-machinery /sig scheduling /assign
this will also need picking to release-1.25
/remove-sig scheduling /sig apps
/assign
/unassign


Question: // ClearUDPConntrackForPeers uses the conntrack tool to delete the conntrack entries // for the UDP connections specified by the {origin, dest} IP pair. func ClearUDPConntrackForPeers(execer exec.Interface, origin, dest string) error { <del> parameters := parametersWithFamily(isIPv6(origin), "-D", "--orig-dst", origin, "--dst-nat", dest, "-p", "udp") </del> <ins> parameters := parametersWithFamily(IsIPv6String(origin), "-D", "--orig-dst", origin, "--dst-nat", dest, "-p", "udp") </ins> err := ExecConntrackTool(execer, parameters...) if err != nil && !strings.Contains(err.Error(), noConnectionToDelete) { // TODO: Better handling for deletion failure. When failure occur, stale udp connection may not get flushed.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: apiserverSigningCert []byte } <ins> var serverAggregatorVersion = utilversion.MustParseSemantic("v1.7.0") </ins> var _ = SIGDescribe("Aggregator", func() { f := framework.NewDefaultFramework("aggregator") framework.AddCleanupAction(func() {

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: } } <del> return pod.Status.Message == podShutdownMessage && pod.Status.Reason == podShutdownReason && hasContainersNotReadyCondition </del> <ins> return pod.Status.Message == podShutdownMessage && pod.Status.Reason == podShutdownReason && hasContainersNotReadyCondition && pod.Status.Phase == v1.PodFailed </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Shutdown a node gracefully. E.g. with . Expected to see pods being evicted, and pods being started on another nodes. Let a deployment create one pod, configure Graceful Node Shutdown as described in , then just that node. Then, the pod will get a state of or , but no new pod is being created and scheduled on a different node. Then, when the node comes back, the terminated pod stays in such a state: Effectively, the pod is running on the node which has been rebooted gracefully, but is in a state. This worked as expected in v1.21 Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig node /cc
/remove-sig api-machinery
As I tested in both 1.21 and 1.22, the big difference was whether pod's is or (Please see details below). I'm suspecting that doesn't change phase to for node shutdown manager (While eviction manager is changing it, ). I'm not familiar with the codes around it, so I'm not sure if it is intended or not, though. <details[k8s 1.21] [k8s 1.22] </details
Note that Graceful Node Shutdown in k8s 1.21 seems to work well even with Statefulset with persistent volumes (Please also see the details below for the console log). It may be out of the scope of this issue, but sharing here just for your information. <details- Deploy statefulset with persistent volumes Rebooting node1 </details
Also, sharing additional test results for "Statefulset with persistent volume" case with k8s 1.21. didn't work well, but worked well (Please also see the details below). Tested with ShutdownGracePeriod=30s, and ShutdownGracePeriodCriticalPods=10s, as described in the blog.

And all the nodes are running on CentOS 8. <details[With ] on node1After manually start the node1 [With ] on node1 </details
Also sharing the results with to the current master branch. I just replaced the kubelet to the patched version on the nodes of k8s 1.22 environment that is previously used. Some of the Pods still remain , but the situation seems improving (Not working perfectly and not patched to the k8s 1.22 release, so I can't say that the patch solved the issue nor the patch did improve). I think that further investigation will be much beyond my current knowledge, so I stop here. I hope that this series of information helps in solving this issue. Feel free to ask me to try another tests with some other configuration. I will keep this environment as they are for a while. <details[Deployment without persistent volume] node2 (Sorry. No pods are scheduled on node1, so I did a bit different test from previous ones.)After node2 is back [StatefulSet with persistent volumes] node2After node2 is back (Or five minutes?) </details
Hi It seems that the node status is not synchronized in time Could you remove the go and try again to see if it can be avoided
/assign
Thank you for your comment. Tested with and confirmed that pods constantly become . As all the pods don't become , they remained in the nodes. <details- After on node1 After node1 is back, again </details
My suspicion is that the graceful shutdown duration of the Pod is the same as that of NodeShutdown. Node shuts down first and the Pod data is not synchronized to ApiServer. Caused the container status not to be updated. Could you try this patch
Commented the test result for the patch in .
/triage accepted /cc
Basically, this was working as expected, at least as expected by documentation in 1.21. Is it not a regression?
This is a regression in 1.22, caused by The change was intentional, since restarting a node should not necessarily stop all pods/containers on that node (see ), but for people depending on the existing behavior, it is definitely a regression.

I think we should do the following: Fix the regression and backport to 1.22.x and 1.23.x to restore the delete-on-shutdown behavior In 1.24, add a kubelet option to permit choosing whether pods are terminated on node shutdown or not and change the default to not terminate. People who want the prior behavior can opt into setting the "terminate-on-shutdown" option in 1.24. This should be done before promoting graceful node shutdown to GA
I agree with Jordan, that is what we should have done. People using graceful shutdown on non-terminal pods should be informed of this behavior change as well (someone deploying in 1.22 not using transient nodes may be impacted by this in a way that causes data loss, for instance if they have data in pod local dirs, and have started using graceful deletion in 1.22 unaware of the previous behavior). Jordan argued that as a beta feature, changing the default in 1.23-1.24 is acceptable, whereas changing the default in a back port fix is unusual, and per our normal behavior with regressions, even with enough of a gap that people might anticipate the new behavior, we try to go back to our previous behavior in patch versions.
see upthread, Jordan and I also briefly discussed back porting the config flag that would set the default as well (we should explore that in the fix), and for distros that want to use beta graceful shutdown but are not using transient nodes, we should recommend that they leverage that flag in order to avoid data loss on pods (if we indeed decide to include the flag).
I dont know if its related, but since 1.22, I frequently see Terminated pods still running. It happens if I accidentally pull the power plug, or something like that. And those pods are up and running after the nodes come back, but their state is Terminated: It seems that these pods are daemonset pods.
The plan in makes sense. I've opened up the following changes to revert to the previous 1.21 behavior. master - 1.23 - 1.22 - We will followup with option to make it configurable in 1.24 as we discussed.


Question: }, ServiceNodePortRange: kubeoptions.DefaultServiceNodePortRange, } <del> s.ServiceClusterIPRanges = kubeoptions.DefaultServiceIPCIDR.String() </del> // Overwrite the default for storage data format. s.Etcd.DefaultStorageMediaType = "application/vnd.kubernetes.protobuf"

API server defaults the service CIDR. We should instead just fail if it is not specified. Defaulting is a wild guess and very likely to be wrong. /sig network /kind bug
For existing clusters that don't currently set it, is setting it to anything other than the current default a breaking change?
Yes. You can not really change it (well, you can expand it, but not totally move it), unless you delete ALL services, including kubernetes.default. I filed this aspirationally. It's not on fire, it's just a dumb thing to have provided a default for.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestPersistentVolumeRecycler(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> deleteAllEtcdKeys() // Use higher QPS and Burst, there is a test for race condition below, which

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: image:    "foo.azurecr.us/bar/image:version", expected: "foo.azurecr.us", }, <ins> { image:    "foo.azurecr.my.cloud/bar/image:version", expected: "foo.azurecr.my.cloud", }, </ins> } for _, test := range tests { <del> if loginServer := parseACRLoginServerFromImage(test.image); loginServer != test.expected { </del> <ins> if loginServer := provider.parseACRLoginServerFromImage(test.image); loginServer != test.expected { </ins> t.Errorf("function parseACRLoginServerFromImage returns "%s" for image %s, expected "%s"", loginServer, test.image, test.expected) } }

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: // headersForConfig extracts any http client logic necessary for the provided // config. <del> func headersForConfig(c *restclient.Config) (http.Header, error) { </del> <ins> func headersForConfig(c *restclient.Config, url *url.URL) (http.Header, error) { </ins> extract := &extractRT{} rt, err := restclient.HTTPWrappersForConfig(c, extract) if err != nil { return nil, err } <del> if _, err := rt.RoundTrip(&http.Request{}); err != nil { </del> <ins> if _, err := rt.RoundTrip(&http.Request{URL: url}); err != nil { </ins> return nil, err } return extract.Header, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When I did e2e conformance test with log level 5, it completely passed! When I did e2e conformance test with log level 8, it just failed. The only thing I did is just leveling up from log level 5 to 8. But, I've got this error message. What you expected to happen: test would be completely passed and just get more verbose log Environment: local cluster <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing
Nice report. I confirmed this issue happens on my local environment also like:


Question: secret, err := kl.secretManager.GetSecret(pod.Namespace, secretRef.Name) if err != nil { klog.InfoS("Unable to retrieve pull secret, the image pull may not succeed.", "pod", klog.KObj(pod), "secret", klog.KObj(secret), "err", err) <ins> failedPullSecrets = append(failedPullSecrets, secretRef.Name) </ins> continue } pullSecrets = append(pullSecrets, *secret) } <ins> if len(failedPullSecrets) > 0 { kl.recorder.Eventf(pod, v1.EventTypeWarning, "FailedToRetrieveImagePullSecret", "Unable to retrieve some image pull secrets (%s); attempting to pull the image may not succeed.", strings.Join(failedPullSecrets, ", ")) } </ins> return pullSecrets }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --imagePullSecrets can to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error. I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace. If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place. Add to a pod spec, and pull images from
/sig node (based on the field location)
(and/or) /sig auth
correct me if not. Pulling private images with nonexists secret, pod will turn to . However, pulling public image with nonexists secret, no error will display yet.
Similar issue with .
Yes that's correct.
I think so. Working on this later. Will also live up to your expectations, right?
But I check it already have a warnging events if secret not exist and pullImageFail: line:147
Now, only info message when pullsecret not exist
maybe direct return and not enter is better. if pullSecret is not existed
if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.
I think a warning event like could be here /good-first-issue /triage accepted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I would like work on this. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work in this. /assign
Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.
Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective. An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod
hi , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not.
To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.
gotcha, that seems more reasonable, thanks for clarifying as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed .

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.
/assign
based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right?
I think has created a above and labeled /lgtm already.
Hey I don't know anything about how to code but i am interested to contribute on kubernetes So help me to contribute
/cc
there is PR already here , just needs approval
it was just FYI =) Reviewed the PR, thank you
Thanks for the review :)
/assign
will like to work on it. /assign
Hey , would love to work on this issue
there is already a PR: do you plan to push the PR forward?
I have been busy with other things recently, if anyone is willing to continue this problem. I will close the PR.
if noone else is working on this task, I would like to take this up.
Feel free to work on this , you can refer to for some advices.
Thanks! Will refer it
/assign
/assign
/assign
Is this issue still open/active?
Hello I was working on it but not able to continue it due to some personal reasons. You can take over if you would like to. You can refer this PR if you want to.
Yes sure!!!
Can I try this if no one is available to work on it ?
Yes go on!!!
/assign
Sorry for keep opening the PR. something went wrong on my fork branch. I eventually needed to re-fork the repo so it caused the PR closed automatically
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Can I give it a go if no one is working on it?
Thanks for the offer, but it looks like was open and had an update that was waiting for review (which I didn't realize)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
is this issue still available to pick up?
/remove-lifecycle stale
Hi, is this issue still available?
Hi! It feels that this issue is stalled (although it lacks the label) and the related PRs have been abandoned. I'll work on it if that's ok /assign
As well as an event, we might like to directly report a warning to end users (suggested in ). More complexity though.
should it be part of this issue? it only mentions but as you guys (the contributors) decide. is based on which was labelled as and suggested as a starting point before being closed.


Question: return true, nil } <ins> func (ctrl *PersistentVolumeController) findPodsByPVCKey(key string) ([]*v1.Pod, error) { pods := []*v1.Pod{} objs, err := ctrl.podIndexer.ByIndex(pvcKeyIndex, key) if err != nil { return pods, err } for _, obj := range objs { pod, ok := obj.(*v1.Pod) if !ok { continue } pods = append(pods, pod) } return pods, err } </ins> // isVolumeUsed returns list of pods that use given PV. func (ctrl *PersistentVolumeController) isVolumeUsed(pv *v1.PersistentVolume) ([]string, bool, error) { if pv.Spec.ClaimRef == nil { return nil, false, nil } <del> claimName := pv.Spec.ClaimRef.Name </del> podNames := sets.NewString() <del> pods, err := ctrl.podLister.Pods(pv.Spec.ClaimRef.Namespace).List(labels.Everything()) </del> <ins> pvcKey := fmt.Sprintf("%s/%s", pv.Spec.ClaimRef.Namespace, pv.Spec.ClaimRef.Name) pods, err := ctrl.findPodsByPVCKey(pvcKey) </ins> if err != nil { <del> return nil, false, fmt.Errorf("error listing pods: %s", err) </del> <ins> return nil, false, fmt.Errorf("error finding pods by pvc %q: %s", pvcKey, err) </ins> } for _, pod := range pods { <del> if util.IsPodTerminated(pod, pod.Status) { continue } for i := range pod.Spec.Volumes { usedPV := &pod.Spec.Volumes[i] if usedPV.PersistentVolumeClaim != nil && usedPV.PersistentVolumeClaim.ClaimName == claimName { podNames.Insert(pod.Namespace + "/" + pod.Name) } } </del> <ins> podNames.Insert(pod.Namespace + "/" + pod.Name) </ins> } return podNames.List(), podNames.Len() != 0, nil } <ins> // findNonScheduledPodsByPVC returns list of non-scheduled pods that reference given PVC. func (ctrl *PersistentVolumeController) findNonScheduledPodsByPVC(pvc *v1.PersistentVolumeClaim) ([]string, error) { pvcKey := fmt.Sprintf("%s/%s", pvc.Namespace, pvc.Name) pods, err := ctrl.findPodsByPVCKey(pvcKey) if err != nil { return nil, err } podNames := []string{} for _, pod := range pods { if len(pod.Spec.NodeName) == 0 { podNames = append(podNames, pod.Name) } } return podNames, nil } </ins> // doDeleteVolume finds appropriate delete plugin and deletes given volume, returning // the volume plugin name. Also, it returns 'true', when the volume was deleted and // 'false' when the volume cannot be deleted because the deleter is external. No

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: framework.Skipf("Environment does not support getting controller-manager metrics - skipping") } <ins> ginkgo.By("Getting plugin name") defaultClass, err := c.StorageV1().StorageClasses().Get(defaultScName, metav1.GetOptions{}) framework.ExpectNoError(err, "Error getting default storageclass: %v", err) pluginName := defaultClass.Provisioner </ins> controllerMetrics, err := metricsGrabber.GrabFromControllerManager() framework.ExpectNoError(err, "Error getting c-m metrics : %v", err) <del> storageOpMetrics := getControllerStorageMetrics(controllerMetrics) </del> <ins> storageOpMetrics := getControllerStorageMetrics(controllerMetrics, pluginName) </ins> pvc, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Create(pvc) framework.ExpectNoError(err)

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -


Question: plugin.host = host plugin.syncNetworkConfig() <ins> // start a goroutine to sync network config from confDir periodically to detect network config updates in every 5 seconds go wait.Forever(plugin.syncNetworkConfig, defaultSyncConfigPeriod) </ins> return nil }

What happened: When Node's iops is full, the Node status becomes to be NotReady. The NotReady message is "Container runtime is not ready". After our troubleshoot, we found func in cni package costs more then 30s, then the runtime status health check is failed : What you expected to happen: Node should still be in Ready status when iops is full. Because the CNI config files does not change. How to reproduce it (as minimally and precisely as possible): Run an application to fill disk IO bandwidth, then the Node becomes to be NotReady. Anything else we need to know?: Nothing yet. Environment: Runtime is dockershim. Other environment is no need to pay attention.
/assign /sig-node
/sig node /sig network


Question: glog.Errorf("Error occurred during reconstruct volume from disk: %v", err) } } <ins> if len(volumeNeedReport) > 0 { rc.desiredStateOfWorld.MarkVolumesReportedInUse(volumeNeedReport) } </ins> } func (rc *reconciler) cleanupMounts(volume podVolume) {

/kind bug What happened: kubelet logs lot of when it starts and a pod was deleted while it was offline. How to reproduce it (as minimally and precisely as possible): a pod with AWS EBS volume (should work also with GCE or any other volume) kubelet the pod kubelet It's not always reproducible, is there some sort of race? My current theory: desiredStateOfWorldPopulator scans pods and adds this pod to DSW reconciler.syncStates() runs. it finds mounted volume but it does not add it to actual state of world because it's in DSW: When kubelet fully starts, it kills the pod and cleans DSW, but the volume never reached AWS so nothing is unmounted. Environment: Kubernetes version (use ): today's master I have troubles reproducing this on master. /sig storage /sig node
Ok, check at won't add the volume to ASW. But the volume is in DSW and subsequent reconcile() should check that the volume is attached and mounted and add it to ASW. This does not happen, because: Volume is not in because it was overwritten by new kubelet (?)
This problem goes beyond a simple logging error. The problem also is - such volumes never get unmounted correctly and hence volume can not be used anywhere. IMO this is kind of serious. Steps to reproduce: a pod on node X kubelet the pod. The pod will be stuck in "Terminating" state back the kubelet There is a race which is rather easy to reproduce and what happens is - volume gets to DSW at kubelet but it never makes into ASW because volume is indeed never in use on node (because pod was deleted while kubelet was down). This results in volume never getting unmounted and hence any attempt to use the volume on different node fails.
IMO, this happens: is called only once at kubelet startup and at that point the pod still exists (it's in Terminating state). The volume is in DSW and it's not to ASW. is called periodically. It could add the volume to ASW, but it does not, because the volume is not in yet. kills pod containers and confirms it's deleted. The pod is deleted from API server. removes the pod from DSW is called and the volume is not there because it's not in DSW.

We have a mounted volume that's not in DSW nor in ASW. There are several possible solutions: a. kubelet can add the volume to ASW in regardless of DSW. b. kubelet can update node status earlier, just after . Will there still be race when pod is deleted during and thus it won't reach ? c. kubelet could call regularly. I am inclined to a), it looks robust to me and it just reflects reality in ASW. It should not matter if the volume is in DSW or not at this point - will sort it out and do the right thing.
solution d.: in reconciler.syncStates() try to reconstruct volume from DSW, fall back to only if the volume is not there. DSW already has spec and other useful info.
Experimental fix (solution d.):
FYI, this might be another 1.10 (or 1.10.1) candidate. We need thorough review though, volumeManager is quite complicated.
another simpler option is perhaps to mark a volume as "ReportedInUse" if it was found on the disk during reconstruction phase. That will make sure that, volume gets correctly to ASW via reconciler(rather than us doing it on reconstruction) and it will make sure that volume is unmounted when volume does get removed from DSW.
I closed , it would blow up when kubelet was restarted at wrong time when creating a pod. Interesting idea, I opened with it. I am not sure it covers all the corner cases though. Newly started kubelet does not overwrite its own status until runs, I hope it's ensured by something and not just a coincidence.
I think might still hit a race condition issue as below is normally deleted (without force delete) during kubelet restart restarts, pod is received again so it is add to the pod Manager is to desired state, so reconciler skip to reconstruct to the actual state kill containers and then try to delete the Pod from API server. If actual state has not yet updated from reconciler, pod will be deleted from pod manager and desired state too will reconcile actual state and desired state. But if step 4 happened first, it will not be able to update actual state since volume is removed from desired state.

Depending on which happens first between step 4 and 5, volume might or might not have a chance to get into the actual state. I am thinking seems better to solve the issue with some additional protection. The idea is to avoid letting kubelet to delete pod before actual state has updated the volume. We can set a flag in actual state to indicate when reconstruction finishes.
what hook you are planning to use to stop kubelet from deleting pod unless it has been processed into asw?
My thought is to add a flag (e.g., finishReconstruct) in actualstateofworld to indicate whether reconstruction finished or not. volumemanager will add a function (e.g., IsActualStateReconstructed) to check the flag is set or not. When call canBeDeleted() function, the PodResourcesAreReclaimed in will return false if actual state has not yet finished reconstructing.
Talked with offline, come up with a solution which combines both and . The proposal still follows the idea of pod is to desired state (no matter whether it might be removed soon or not), do not reconstruct directly (reconstruction does not has the full information which might fail or cause problem having wrong information since there is not enough checking), instead reconcile should have a chance to reconcile actual state by going through the volume set up process. if the pod was force deleted from API server during kubelet down, reconciler tries to clean up mounts directly if possible. The main problem currently is that pod might be deleted from desired state before reconciler updates the actual state. is trying to prevent of this. makes sure that reconciler can update the actual state as soon as possible.
please add this to 1.10 milestone. This should be a 1.10 blocker.
SIG-leads, please confirm, thanks!
/priority critical-urgent /status in-progress /kind bug
You must be a member of the github team to add status labels.
Adding to 1.10 as blocker per
/status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I spoke with offline. The volume reconstruction work (PR ) is non-critical but touches critical mount/unmount code paths. While the PRs that and have ( and ) may resolve the issue identified here, I'm afraid there may be other regressions introduced that we are not aware of. Therefore, I'm asking Jing to revert the volume reconstruction PR. We can reintroduce it along with the patches early in 1.11 along with more extensive testing. Unfortunately, because changes went in on top, an automatic revert is not possible. Jing is working on a manual revert. could you please help test her revert PR when it is ready?
Update on this issue: spent the morning trying to create a revert PR , but it turns out a couple of other big refactors (including ) went in on top of her volume reconstruction PR, making it much more difficult to revert. At this point says she can either spend more time backing out her changes and, in doing so, possibly introduce new bugs or spend time making sure the proposed fixes are rock solid. She says she feels confident that the bugs are understood, and the proposed PRs fixing it should be sufficient, and would like to spend time to double and triple checking it with instead. So and will proceed with this.
i support this approach
I support this as well. Thank you and for working on this.
okay I integrated both the PRs in a branch and tested both on AWS and GCE . Ran e2e tests as well and it looks all good.
I lied about e2e test run. A second run surfaced some failures:
the error message indicates the pods failed to enter running condition. I think my fix should not affect pod creation.
yeah I agree. I noticed that these tests use NFS based PVs which are not ideal source for testing the bug we are trying to reproduce. In this case the NFS server itself may not start correctly and hence the test may fail.

I am looking into refactoring the test suite, so as it can use GCE/EBS sources as well.
I created a follow up issue for creating the disruptive e2e tests so that bugs like these can be more easily caught - . Will follow up with PR tomorrow. Also ran e2e test after removing NFS part and just relying on GCE-PD and it all passed. I am going to let run couple of hours in a loop and see if there is a regression.
thanks for your work here. How did it come out?
we still have some problems that needs to be addressed for . I will sync up with when online.
Perfect, thank you.
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can we keep it open, we still need the second PR
/cc
re-opening per
The second PR got merged late night yesterday as well. So we can close this. /close


Question: return acc.GetResourceVersion(), nil } <ins> func TestNoopChangeCreationTime(t *testing.T) { client, closeFn := setup(t) defer closeFn() ssBytes := []byte(`{ "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "myconfig", "creationTimestamp": null, "resourceVersion": null }, "data": { "key": "value" } }`) obj, err := client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource("configmaps"). Name("myconfig"). Body(ssBytes). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } require.NoError(t, err) // Sleep for one second to make sure that the times of each update operation is different. time.Sleep(1200 * time.Millisecond) newObj, err := client.CoreV1().RESTClient().Patch(types.ApplyPatchType). Namespace("default"). Param("fieldManager", "apply_test"). Resource("configmaps"). Name("myconfig"). Body(ssBytes). Do(context.TODO()). Get() if err != nil { t.Fatalf("Failed to create object: %v", err) } require.NoError(t, err) require.Equal(t, obj, newObj) } </ins> // TestNoSemanticUpdateAppleSameResourceVersion makes sure that APPLY requests which makes no semantic changes // will not change the resource version (no write to etcd is done) //

Sending a patch built from a typed client to SSA will force a write to etcd (and bump resourceVersion) since creationTimestamp is serialized to null. This is a follow-up of which doesn't solve the issue properly. Sending a no-op SSA request with creationTimestamp to null shouldn't do anything Kubectl apply an object with creationTimestamp null twice and see that it updates the object. Probably something like this would cause it:
/sig api-machinery /wg api-expression /triage accepted /assign /priority important-soon


Question: close(stopCh) } <ins> func TestTransformingInformerRace(t *testing.T) { // source simulates an apiserver object endpoint. source := fcache.NewFakeControllerSource() label := "to-be-transformed" makePod := func(name string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: "namespace", Labels:    map[string]string{label: "true"}, }, Spec: v1.PodSpec{ Hostname: "hostname", }, } } badTransform := atomic.Bool{} podTransformer := func(obj interface{}) (interface{}, error) { pod, ok := obj.(*v1.Pod) if !ok { return nil, fmt.Errorf("unexpected object type: %T", obj) } if pod.ObjectMeta.Labels[label] != "true" { badTransform.Store(true) return nil, fmt.Errorf("object already transformed: %#v", obj) } pod.ObjectMeta.Labels[label] = "false" return pod, nil } numObjs := 5 for i := 0; i < numObjs; i++ { source.Add(makePod(fmt.Sprintf("pod-%d", i))) } type event struct{} events := make(chan event, numObjs) recordEvent := func(eventType watch.EventType, previous, current interface{}) { events <- event{} } checkEvents := func(count int) { for i := 0; i < count; i++ { <-events } } store, controller := NewTransformingInformer( source, &v1.Pod{}, 5*time.Millisecond, ResourceEventHandlerDetailedFuncs{ AddFunc:    func(obj interface{}, isInInitialList bool) { recordEvent(watch.Added, nil, obj) }, UpdateFunc: func(oldObj, newObj interface{}) { recordEvent(watch.Modified, oldObj, newObj) }, DeleteFunc: func(obj interface{}) { recordEvent(watch.Deleted, obj, nil) }, }, podTransformer, ) stopCh := make(chan struct{}) go controller.Run(stopCh) checkEvents(numObjs) // Periodically fetch objects to ensure no access races. wg := sync.WaitGroup{} errors := make(chan error, numObjs) for i := 0; i < numObjs; i++ { wg.Add(1) go func(index int) { defer wg.Done() key := fmt.Sprintf("namespace/pod-%d", index) for { select { case <-stopCh: return default: } obj, ok, err := store.GetByKey(key) if !ok || err != nil { errors <- fmt.Errorf("couldn't get the object for %v", key) return } pod := obj.(*v1.Pod) if pod.ObjectMeta.Labels[label] != "false" { errors <- fmt.Errorf("unexpected object: %#v", pod) return } } }(i) } // Let resyncs to happen for some time. time.Sleep(time.Second) close(stopCh) wg.Wait() close(errors) for err := range errors { t.Error(err) } if badTransform.Load() { t.Errorf("unexpected transformation happened") } } </ins> func TestDeletionHandlingObjectToName(t *testing.T) { cm := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{

We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used.

FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted


Question: "sync" ) <del> // WellKnownRegions is the complete list of regions known to the AWS cloudprovider </del> <ins> // wellKnownRegions is the complete list of regions known to the AWS cloudprovider </ins> // and credentialprovider. <del> var WellKnownRegions = [...]string{ </del> <ins> var wellKnownRegions = [...]string{ </ins> // from `aws ec2 describe-regions --region us-east-1 --query Regions[].RegionName | sort` "ap-northeast-1", "ap-northeast-2",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: #   KUBE_MANIFESTS_TAR function find-release-tars() { SERVER_BINARY_TAR=$(find-tar kubernetes-server-linux-amd64.tar.gz) <ins> if [[ -z "${SERVER_BINARY_TAR}" ]]; then exit 1 fi </ins> if [[ "${NUM_WINDOWS_NODES}" -gt "0" ]]; then NODE_BINARY_TAR=$(find-tar kubernetes-node-windows-amd64.tar.gz) fi

What happened: If doesn't exist in $KUBEROOT/output/release-tars, the script doesn't prompt for downloading from external release URL, but just move on util an error is thrown instead. What you expected to happen: If the tarball does not exist, it should be able to download from external release URL. How to reproduce it (as minimally and precisely as possible): remove the tarball from $KUBEROOT/output/release-tars run Anything else we need to know?: The script has everything there, but there is a minor issue has caused the script move on silently after the close the child function. Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
tagging with the appropriate sigs, but please mind that kube-up is deprecated and should not be used in the wild. please move to a different tool for cluster creation, like kubeadm, kops, minikube, kind, cluster-api etc. /sig cluster-lifecycle testing


Question: func (udp *udpProxySocket) ProxyLoop(service proxy.ServicePortName, myInfo *serviceInfo, proxier *Proxier) { var buffer [4096]byte // 4KiB should be enough for most whole-packets for { <del> if info, exists := proxier.getServiceInfo(service); !exists || info != myInfo { </del> <ins> if !myInfo.isAlive() { </ins> // The service port was closed or replaced. break }

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: return err } e2elog.Logf("HugePages_Total is set to %v", numHugePages) <del> if numHugePages == 50 { </del> <ins> if numHugePages == 5 { </ins> return nil } <del> return fmt.Errorf("expected hugepages %v, but found %v", 50, numHugePages) </del> <ins> return fmt.Errorf("expected hugepages %v, but found %v", 5, numHugePages) </ins> } // releaseHugePages releases all pre-allocated hugepages

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: -tests Since when has it been failing: Unsure Testgrid link: Reason for failure: Unable to pre allocate 50x2MiB huge pages, probably due to memory pressure caused by kernel level caching. If requesting an substantial amount of huge page memory (like the tests), it is recommended to to it as early after boot as possible, or by using the kernel cmd line. Eg. this one: Anything else we need to know: will fix this issue, but in awaiting for that, we can just decrease the page count in the tests.
/sig node /cc derekwaynecarr
I think the reason of railure should be because a single hugepage of x86 is 2MiB as and a single page is 4KiB, not 2KiB.
Yes, is correct. Thanks for catching the typo :smile:
Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. This is probably because the kernel is caching a lot of stuff, making it hard for the kernel to fix it. Ill do some testing with dropping the caches (no user facing stuff will happen, only kernel level) before trying to pre allocate huge pages. /reopen
Reopened this issue. <detailsIn response to : Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM BASEIMAGE # The samba-common, cifs-utils, and nfs-common packages depend on # ucf, which itself depends on /bin/bash existing. # It doesn't seem to actually need bash, however. RUN ln -s /bin/sh /bin/bash RUN echo CACHEBUST>/dev/null && clean-install  iptables  ebtables  ethtool  ca-certificates  conntrack  util-linux  socat  git  jq  nfs-common  glusterfs-client  cifs-utils  ceph-common COPY cni-bin/bin /opt/cni/bin </ins>

I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...


Question: import ( "io/ioutil" "os" <ins> "os/user" </ins> "path/filepath" "strings" "testing"

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: TestNsenterExistsFile test is failing during make test What you expected to happen: The test should have passed. How to reproduce it (as minimally and precisely as possible): git clone run make test Anything else we need to know?: Fail logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig storage
If you run the test with root user, It will fail. I have make a fix:
thanks for the patch .. its working.


Question: assert.Equal(t, machineInfo, &cadvisorapi.MachineInfo{ NumCores:       4, MemoryCapacity: 1.6e+10, <del> MachineID:      "somehostname"}) </del> <ins> MachineID:      "somehostname", SystemUUID:     "E6C8AC43-582B-3575-4E1F-6DA170888906"}) </ins> } func TestWinVersionInfo(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The windows kubelet does not set .status.nodeInfo.systemUUID on the node object. What you expected to happen: The windows kubelet sets the systemUUID from How to reproduce it (as minimally and precisely as possible): Start the windows kubelet and check for UUID: Anything else we need to know?: This will affect users running the out-of-tree vsphere CCM since that depends on the systemUUID, unlike the in-tree version. Also, looks like cAdvisor is the component responsible for providing the systemUUID so let me know if this issue needs to be moved to that repo. Environment: Kubernetes version (use ): Verified on 1.13.1 and 1.14 Cloud provider or hardware configuration: None OS (e.g: ): Windows Server 2019
/sig windows /sig vmware
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <ins> # Copyright 2018 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. .PHONY: manifest-tool MANIFEST_TOOL_DIR := $(shell mktemp -d) export PATH := $(MANIFEST_TOOL_DIR):$(PATH) MANIFEST_TOOL_VERSION := v0.7.0 space := space += comma := , prefix_linux = $(addprefix linux/,$(strip $1)) join_platforms = $(subst $(space),$(comma),$(call prefix_linux,$(strip $1))) manifest-tool: curl -sSL https://github.com/estesp/manifest-tool/releases/download/$(MANIFEST_TOOL_VERSION)/manifest-tool-linux-amd64 > $(MANIFEST_TOOL_DIR)/manifest-tool chmod +x $(MANIFEST_TOOL_DIR)/manifest-tool </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue if for tracking the work related to pushing manifest list for kubernetes images(infra and test) List of images for which manifest need to be created: - [x] kubernetes to use manifest pause image - [x] dns - - - - - [ ] dashboard [x] Manifest images for k8s release images -
/assign
/cc
/sig cluster-lifecycle
/sig testing
/area test-infra /sig release
/reopen
I'd add dashboard to this as well, based off the issue at
can you provide a status update on this issue? I'd really want this fully working in v1.10... :)
I've created one more PR for pushing manifest for few more images, reference - I'll try my best to push into v1.10 release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle stale
/remove-lifecycle rotten
A problem related to manifests and was addressed here:
An update of this issue would be welcomed to track the status of all affected images, since I think at least some of the linked issues have progressed.
/sig testing
all the k8s core images are now fat manifest, hence closing this task. Will create a separate task for creating manifest for ecosystem images(e.g dashboard, metric-server etc...)
did you have a issue open tracking other images? I noticed today that kube-addon-manager (cluster/addons/addon-manager) is multi-arch but not using a manifest list yet.


Question: all-container: $(addprefix sub-container-,$(ALL_ARCH)) <del> all-push: $(addprefix sub-push-,$(ALL_ARCH)) </del> <ins> all-push-images: $(addprefix sub-push-,$(ALL_ARCH)) </ins> build: bin/$(BIN)-$(ARCH)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue if for tracking the work related to pushing manifest list for kubernetes images(infra and test) List of images for which manifest need to be created: - [x] kubernetes to use manifest pause image - [x] dns - - - - - [ ] dashboard [x] Manifest images for k8s release images -
/assign
/cc
/sig cluster-lifecycle
/sig testing
/area test-infra /sig release
/reopen
I'd add dashboard to this as well, based off the issue at
can you provide a status update on this issue? I'd really want this fully working in v1.10... :)
I've created one more PR for pushing manifest for few more images, reference - I'll try my best to push into v1.10 release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle stale
/remove-lifecycle rotten
A problem related to manifests and was addressed here:
An update of this issue would be welcomed to track the status of all affected images, since I think at least some of the linked issues have progressed.
/sig testing
all the k8s core images are now fat manifest, hence closing this task. Will create a separate task for creating manifest for ecosystem images(e.g dashboard, metric-server etc...)
did you have a issue open tracking other images? I noticed today that kube-addon-manager (cluster/addons/addon-manager) is multi-arch but not using a manifest list yet.


Question: ) var _ = SIGDescribe("Kubernetes Dashboard", func() { <ins> BeforeEach(func() { // TODO(kubernetes/kubernetes#61559): Enable dashboard here rather than skip the test. framework.SkipIfProviderIs("gke") }) </ins> const ( uiServiceName = "kubernetes-dashboard" uiAppName     = uiServiceName

Is this a BUG REPORT or FEATURE REQUEST?: What happened: GKE tests are failing. They are trying to check that the dashboard is alive, but the dashboard is disabled by default in GKE in Kubernetes 1.10. What you expected to happen: The test should either not run, or start the dashboard. Ideally, it would start the dashboard so we could see that it works, for now we'll just disable the test. How to reproduce it (as minimally and precisely as possible): https://k8s- Anything else we need to know?: Environment: Kubernetes version (use ): 1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: cc
/priority critical-urgent That's probably not true, but please change it to appropriate.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Is this different from ?
we have enabled the plugin from all the blocking jobs so it's not a blocker, but in the long run this need to be in it's own feature.


Question: var _ = SIGDescribe("Security Context", func() { f := framework.NewDefaultFramework("security-context-test") <del> f.NamespacePodSecurityEnforceLevel = admissionapi.LevelBaseline </del> <ins> f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged </ins> var podClient *framework.PodClient ginkgo.BeforeEach(func() { podClient = f.PodClient()

E2eNode Suite: [sig-node] Security Context when creating a pod in the host PID namespace should show its pid in the host PID namespace [NodeFeature:HostAccess] expand_less since last night. NA /sig auth node
This is failing on a ton of jobs. /kind regression /priority important-soon /milestone v1.24 Triage dump:
/triage accepted


Question: "time" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/wait"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!