Question: status = &podSyncStatus{ syncedAt: now, } <ins> // if this pod is being synced for the first time, we need to make sure it is an active pod if !isRuntimePod && (pod.Status.Phase == v1.PodFailed || pod.Status.Phase == v1.PodSucceeded) { // check to see if the pod is not running and the pod is terminal. // If this succeeds then record in the podWorker that it is terminated. if statusCache, err := p.podCache.Get(pod.UID); err == nil { if isPodStatusCacheTerminal(statusCache) { status = &podSyncStatus{ terminatedAt:       now, terminatingAt:      now, syncedAt:           now, startedTerminating: true, finished:           true, } } } } </ins> p.podSyncStatuses[uid] = status }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Restart: Always pods went into OutOfCpu error state after kubelet restart. Pods should remain running as node had sufficient capacity for them. Completed, RestartNever pod should not revert from Completed to OutOfCpu. Launch kubelet with: Run the following script against the local cluster: <details</detailsObserve pod status: Restart the kubelet: Observe pods: xref /sig node Kubernetes version (use ): v1.23.0-alpha.3.183+ - Cloud provider or hardware configuration: Local machine OS (e.g: ) & Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others:
/triage accepted /priority critical-urgent Reproducible on 1.22
From the reproducer: yields However, in the kubelet logs, it sees 7100m cpu allocated?
Looks like the topology admit handler thinks that the completed pods are using CPU which is why when it gets to the 8th it fails it with OutOfCpu. Must be a bug in there somewhere.
/assign
Unclear if could be related but linking for posterity
Confirmed this only affects 1.22, repro on 1.21 has expected behaviour: /sig scheduling I'll start a bisect.
is
Added some logging With this we see I think that the terminated pods are getting filtered, but for whatever reason they are still getting their resource requests to NodeInfo?
points out that my logging language was a bit backwards and that the completed pods should not have been returned, because they already successfully completed and exited. This code is suspect: The logs show that the completed pods are never filtered, so they are failing one of these checks. I am going to try to write a test case that hits this issue so we can test fixes against it.

The completed pods definitely show up as complete, but for some reason they do not get filtered (note I have changed my patch above for the log lines to make sense now): So I guess isPodTerminationRequested=true on already exited pods, but I would only expect that to be true if the pod was actively in the process of termination and not already exited. cc Log patch: <details</details
I tried changing to but apparently it thinks that's true :( Edit: doesn't work either.
The bug is that the only component in the kubelet that is allowed to authoritatively state a container is terminated (pod worker) takes non-zero time to do so during startup (because we have to call kill container to guarantee it's stopped). This races with the pod config loop and admission, which sends all the containers at once. In order for the race to be eliminated, some equivalent of a sync has to be completed during startup on terminal but potentially non-terminated pod workers BEFORE other adds are processed. That could be running HandlePodCleanups once all sources are synced but before HandlePod* methods are called, but HandlePodAdditions() cannot be invoked until after the pod worker has had a chance to verify all terminal pods are actually terminal. EDIT: Another potential approach is for the pod worker to hold starting pods until SyncKnownPods observes all terminal pods are accounted for at least once, then allowing starts.
Also, we appear to be missing a kubelet restart test that verifies admission with completed pods (which I thought was on our list of "missing e2e tests that would have caught these regressions" already). Such a test would have to: pods that consume resources for admission until they are running some of them with failed and` successful states the kubelet some of the running pods the kubelet none of the terminal pods are restarted, all running pods are restarted, and that only non-terminal pods are accounted to admission


Question: ) evictionsNumber = metrics.NewCounterVec( &metrics.CounterOpts{ <ins> Subsystem:         nodeControllerSubsystem, Name:              evictionsNumberKey, Help:              "Number of Node evictions that happened since current instance of NodeController started, This metric is replaced by node_collector_evictions_total.", DeprecatedVersion: "1.24.0", StabilityLevel:    metrics.ALPHA, }, []string{"zone"}, ) evictionsTotal = metrics.NewCounterVec( &metrics.CounterOpts{ </ins> Subsystem:      nodeControllerSubsystem, <del> Name:           evictionsNumberKey, </del> <ins> Name:           evictionsTotalKey, </ins> Help:           "Number of Node evictions that happened since current instance of NodeController started.", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"zone"}, )

I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted


Question: // Test purposeful shutdown go manager.watchControllers() <del> time.Sleep(10 * time.Millisecond) </del> <ins> fakeEtcd.WaitForWatchCompletion() </ins> fakeEtcd.WatchStop <- true <del> time.Sleep(10 * time.Millisecond) </del> // Did everything shut down? if _, open := <-fakeEtcd.WatchResponse; open {

Detected by
Create a Pod as follows: Name: jenkins Using image: jenkins In a new Kubenetes namespace named website-frontend kubectl create ns website-frontend apiVersion: v1 kind: Pod metadata: name: jenkins namespace: website-frontend spec: containers: name: jenkins image: jenkins kubectl create -f


Question: "k8s.io/client-go/tools/cache" "k8s.io/klog/v2" podutil "k8s.io/kubernetes/pkg/api/v1/pod" <ins> "k8s.io/kubernetes/pkg/apis/core/validation" </ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme" "k8s.io/kubernetes/pkg/scheduler/core"

What happened: When a pod can't be scheduled and the cardinality of taints on nodes is high, the Note field in the event can become bigger than the 1024 allowed and the event isn't sent. The pod stays in a state of without en event which makes the User Experience very bad What you expected to happen: I would expect the Kube-scheduler to be able to send the event no matter how many different taints are on nodes How to reproduce it (as minimally and precisely as possible): add many different taints to nodes and then trying to schedule something which can't be successfully scheduled (higher CPU/memory than any node has) Anything else we need to know?: In this case the length of the message was 1334 characters. Environment: Kubernetes version (use ): 1.18.9 Cloud provider or hardware configuration: bare metal Possible solutions: make event size configurable, it's now hardcoded to 1024 characters make scheduler note aware of that limit and work around I, if length 1024 then truncate the string (so at least something is shown) produce N events splitting the text (preferred one): change event Note to be to something like: (the whole complete information would still be present in the pod status field. for debugging purposes)
/sig scheduling
/triage accepted /cc
I could provide the fix in case we define which solution is best
This is a consequence of I think we can just drop some reasons when we are building the status message if the message is too large. This comes from here /help
/remove-help /assign
how many different taints did you have? I think in all cases we should truncate the message as well:
Another remidiation is to supress outoutping nil fields (such as GenerateName:"", SelfLink:"", UID:"", ResourceVersion:"", etc.) in the EventRecorder side.
The size limit we're hitting is specifically on the field
Completely agreed
good issue~
/assign
/unassign


Question: // Deploys `allocatablePods + 1` pods, each with a memory limit of `1/allocatablePods` of the total allocatable // memory, then confirms that the last pod failed because of failedScheduling func overrideAllocatableMemoryTest(f *framework.Framework, allocatablePods int) { <del> const ( podType = "memory_limit_test_pod" ) totalAllocatable := getTotalAllocatableMemory(f) memValue := totalAllocatable.Value() memPerPod := memValue / int64(allocatablePods) ginkgo.By(fmt.Sprintf("Deploying %d pods with mem limit %v, then one additional pod", allocatablePods, memPerPod)) // these should all work pods := newMemLimitTestPods(allocatablePods, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().CreateBatch(pods) failurePods := newMemLimitTestPods(1, imageutils.GetPauseImageName(), podType, strconv.FormatInt(memPerPod, 10)) f.PodClient().Create(failurePods[0]) gomega.Eventually(func() bool { eventList, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err) for _, e := range eventList.Items { // Look for an event that shows FailedScheduling if e.Type == "Warning" && e.Reason == "FailedScheduling" && e.InvolvedObject.Name == failurePods[0].ObjectMeta.Name { framework.Logf("Found %+v event with message %+v", e.Reason, e.Message) return true } } return false }, 3*time.Minute, 10*time.Second).Should(gomega.Equal(true)) } // newMemLimitTestPods creates a list of pods (specification) for test. func newMemLimitTestPods(numPods int, imageName, podType string, memoryLimit string) []*v1.Pod { var pods []*v1.Pod memLimitQuantity, err := resource.ParseQuantity(memoryLimit) </del> <ins> selector := labels.Set{"kubernetes.io/os": "windows"}.AsSelector() nodeList, err := f.ClientSet.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{ LabelSelector: selector.String(), }) </ins> framework.ExpectNoError(err) <del> for i := 0; i < numPods; i++ { podName := "test-" + string(uuid.NewUUID()) pod := v1.Pod{ </del> <ins> for _, node := range nodeList.Items { status := node.Status podName := "mem-test-" + string(uuid.NewUUID()) pod := &v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ Name: podName, <del> Labels: map[string]string{ "type": podType, "name": podName, }, </del> }, Spec: v1.PodSpec{ <del> // Restart policy is always (default). </del> Containers: []v1.Container{ { <del> Image: imageName, </del> Name:  podName, <ins> Image: imageutils.GetPauseImageName(), </ins> Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ <del> v1.ResourceMemory: memLimitQuantity, </del> <ins> v1.ResourceMemory: status.Allocatable[v1.ResourceMemory], </ins> }, }, },

The test : is failing because it assumes the memory is the same across all nodes. A change in the configuration changed that assumption and the test started to fail. the test to assign the correct amount of memory to each of the nodes to fill them up. create a Windows cluster with 3 nodes. Two have 6gb of allocatable memory, 1 has 14gb of allocatable memory. Run the test above and it will fail because it tries to create 3 pods with 9 gb of memory. The first pod will land on the node with 14gb of memory but the other two pods won't be able to find a node with enough memory (9gb pod 6 gb nodes). fixes this in the configuration /sig windows <details</details<details</details<details</details<details</details<details</details<details</details
/triage accepted /priority important-long-term
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted /priority important-long-term Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please inform that Am I reasonable here or not. We need to update the in all the cases inside the (, and ) similar to what was done in PR, but now we need to update it in all cases. If you could please confirm that I am reasonable here then I can begin to make these changes in a PR.
sorry I could have been clearer in the issue. The test it's self needs to be updated to handle nodes that do not have consistent memory. The logic in the test is wrong if the nodes don't all have the same memory values: This results in pods that won't fit on some of the nodes and leaves the pods pending. The test should handle this situation by making sure pods filled up memory but can be scheduled across the nodes. Let me know of need more clarification.
So, here we have to fix the logic because the logic is wrong for the case when the nodes don't have the same memory values.

Is the node memory value is equal to which gets assigned to variable in ?My understanding of tackling the issue : So, we need to change because it is the total memory and when it is fixed then the pods present will get appropriate memory limit by . So, if updating the logic for the value of is our focus, such that the it (memory limit) is changed according to the nodes, how can we achieve that ? Am I thinking reasonable here? If my understanding is correct till now then how we will be changing the code for assigning the memory value for a node such that each of the nodes get relevant amount of memory based on there need to work properly.
Yes, I think think the simplest thing to so would be get a list of windows nodes iterate though each one, looking up the allocatable memory for the node, creating a pod to fill that memory and assign it to that node for scheduling via , ensuring the scheduling was successful. Then once the nodes are all full deploy the pod that should go into pending (code should stay the same). pseudo code:
Do we have some examples in the repo where we do something similar for other nodes ? First we need to make function or is it already present ? Confirming : Also Do we need to run the whole code(After Modifying) inside in loop for every windows node and if even one return fails, the test fails.
I need some guidance. Could you please take a look on previous comment.
example of getting the nodes and memory total: no, basically need to replace some of the code in that function with the sudo code in
Why are we not deploying both the pods of in one loop only (if we want memory of pending pods memory same as nodes memory). Also after we deploys both the pods, do we will be running another loop similar to for checking the failure. I am not much sure about my changes, can I make a PR so that I can discuss about my code and make appropriate changes further ?
Sounds good
Does functions , and already exists or we have to make them because I am not able to find, have you used these names as short forms for other functions which do same as their names ?


Question: assert.False(t, isRegionValid("pl-fake-991a", fake.metadata), "expected region 'pl-fake-991' to be invalid but it was not") } <del> func TestGetCandidateZonesForDynamicVolume(t *testing.T) { tests := []struct { name          string labels        map[string]string ready         bool expectedZones sets.String }{ { name:          "master node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMasterRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label empty", labels:        map[string]string{labelKeyNodeMaster: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label true", labels:        map[string]string{labelKeyNodeMaster: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString(), }, { name:          "master node with master label false", labels:        map[string]string{labelKeyNodeMaster: "false", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with role label", labels:        map[string]string{labelKeyNodeRole: nodeMinionRole, v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with minion label", labels:        map[string]string{labelKeyNodeMinion: "", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "minion node with compute label", labels:        map[string]string{labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "master and minion node", labels:        map[string]string{labelKeyNodeMaster: "true", labelKeyNodeCompute: "true", v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, { name:          "node not ready", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         false, expectedZones: sets.NewString(), }, { name:          "node has no zone", labels:        map[string]string{}, ready:         true, expectedZones: sets.NewString(), }, { name:          "node with no label", labels:        map[string]string{v1.LabelZoneFailureDomain: "us-east-1a"}, ready:         true, expectedZones: sets.NewString("us-east-1a"), }, } for i, test := range tests { t.Run(test.name, func(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterID) c, _ := newAWSCloud(CloudConfig{}, awsServices) c.kubeClient = fake.NewSimpleClientset() nodeName := fmt.Sprintf("node-%d", i) _, err := c.kubeClient.CoreV1().Nodes().Create(&v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:   nodeName, Labels: test.labels, }, Status: genNodeStatus(test.ready), }) assert.Nil(t, err) zones, err := c.GetCandidateZonesForDynamicVolume() assert.Nil(t, err) assert.Equal(t, test.expectedZones, zones) }) } } func genNodeStatus(ready bool) v1.NodeStatus { status := v1.ConditionFalse if ready { status = v1.ConditionTrue } ret := v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady, Status: status, }, }, } return ret } </del> func newMockedFakeAWSServices(id string) *FakeAWSServices { s := NewFakeAWSServices(id) s.ec2 = &MockedFakeEC2{FakeEC2Impl: s.ec2.(*FakeEC2Impl)}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?

I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) } } <ins> func TestGetHistogramVecFromGatherer(t *testing.T) { var registerMetrics sync.Once tests := []struct { name    string lvMap   map[string]string wantVec HistogramVec }{ { name:  "filter with one label", lvMap: map[string]string{"label1": "value1-0"}, wantVec: HistogramVec{ &Histogram{&dto.Histogram{ SampleCount: uint64Ptr(1), SampleSum:   pointer.Float64Ptr(1.5), Bucket: []*dto.Bucket{ {CumulativeCount: uint64Ptr(0), UpperBound: pointer.Float64Ptr(0.5)}, {CumulativeCount: uint64Ptr(1), UpperBound: pointer.Float64Ptr(2.0)}, {CumulativeCount: uint64Ptr(1), UpperBound: pointer.Float64Ptr(5.0)}, }, }}, &Histogram{&dto.Histogram{ SampleCount: uint64Ptr(1), SampleSum:   pointer.Float64Ptr(2.5), Bucket: []*dto.Bucket{ {CumulativeCount: uint64Ptr(0), UpperBound: pointer.Float64Ptr(0.5)}, {CumulativeCount: uint64Ptr(0), UpperBound: pointer.Float64Ptr(2.0)}, {CumulativeCount: uint64Ptr(1), UpperBound: pointer.Float64Ptr(5.0)}, }, }}, }, }, { name:  "filter with two labels", lvMap: map[string]string{"label1": "value1-0", "label2": "value2-1"}, wantVec: HistogramVec{ &Histogram{&dto.Histogram{ SampleCount: uint64Ptr(1), SampleSum:   pointer.Float64Ptr(2.5), Bucket: []*dto.Bucket{ {CumulativeCount: uint64Ptr(0), UpperBound: pointer.Float64Ptr(0.5)}, {CumulativeCount: uint64Ptr(0), UpperBound: pointer.Float64Ptr(2.0)}, {CumulativeCount: uint64Ptr(1), UpperBound: pointer.Float64Ptr(5.0)}, }, }}, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { buckets := []float64{.5, 2, 5} // HistogramVec has two labels defined. labels := []string{"label1", "label2"} HistogramOpts := &metrics.HistogramOpts{ Namespace: "namespace", Name:      "metric_test_name", Subsystem: "subsystem", Help:      "histogram help message", Buckets:   buckets, } vec := metrics.NewHistogramVec(HistogramOpts, labels) registerMetrics.Do(func() { legacyregistry.MustRegister(vec) }) // Observe two metrics with same value for label1 but different value of label2. vec.WithLabelValues("value1-0", "value2-0").Observe(1.5) vec.WithLabelValues("value1-0", "value2-1").Observe(2.5) vec.WithLabelValues("value1-1", "value2-0").Observe(3.5) vec.WithLabelValues("value1-1", "value2-1").Observe(4.5) metricName := fmt.Sprintf("%s_%s_%s", HistogramOpts.Namespace, HistogramOpts.Subsystem, HistogramOpts.Name) histogramVec, _ := GetHistogramVecFromGatherer(legacyregistry.DefaultGatherer, metricName, tt.lvMap) if diff := cmp.Diff(tt.wantVec, histogramVec); diff != "" { t.Errorf("Got unexpected HistogramVec (-want +got):n%s", diff) } }) } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : run got the below error message, We should provide the metric data for different extension points from the metric of "schedulerframeworkextensionpointdurationseconds" Why is this needed: Those metrics have been removed by this commit - Those metrics are deprecated cause it is suggested to use - , but simply remove those metrics definitely is not a nice way to address the issue. /sig scheduling
/triage accept
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: // Check PV node affinity on bound volumes if len(boundClaims) > 0 { <del> boundVolumesSatisfied, err = b.checkBoundClaims(boundClaims, node, podName) </del> <ins> boundVolumesSatisfied, boundPVsFound, err = b.checkBoundClaims(boundClaims, node, podName) </ins> if err != nil { return }

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: }, { "ImportPath": "github.com/docker/spdystream", <del> "Rev": "e731c8f9f19ffd7e51a469a2de1580c1dfbb4fae" </del> <ins> "Rev": "99515db39d3dad9607e0293f18152f3d59da76dc" </ins> }, { "ImportPath": "github.com/elazarl/go-bindata-assetfs",

Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently?

I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.


Question: // these evaluators require an alias for backwards compatibility for gvr, alias := range legacyObjectCountAliases { result = append(result, <del> generic.NewObjectCountEvaluator(false, gvr.GroupResource(), generic.ListResourceUsingListerFunc(f, gvr), alias)) </del> <ins> generic.NewObjectCountEvaluator(gvr.GroupResource(), generic.ListResourceUsingListerFunc(f, gvr), alias)) </ins> } return result }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: }, Status: corev1.PersistentVolumeClaimStatus{}, }, <del> expectedElements: []string{"DataSource:n  Name:      src-snapshotn  Kind:      VolumeSnapshotn"}, </del> <ins> expectedElements: []string{"DataSource:n  APIGroup:  snapshot.storage.k8s.ion  Kind:      VolumeSnapshotn  Name:      src-snapshotn"}, </ins> }, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Miss APIGroup of DataSource. What you expected to happen: kubectl describe pvc should also describe the apigroup How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage


Question: // SchemeGroupVersion is group version used to register these objects var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: "v1"} <ins> // Unversioned is group version for unversioned API objects // TODO: this should be v1 probably var Unversioned = schema.GroupVersion{Group: "", Version: "v1"} </ins> // WatchEventKind is name reserved for serializing watch events. const WatchEventKind = "WatchEvent"

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: return ret, nil } <ins> func (c *Cloud) addLoadBalancerTags(loadBalancerName string, requested map[string]string) error { var tags []*elb.Tag for k, v := range requested { tag := &elb.Tag{ Key:   aws.String(k), Value: aws.String(v), } tags = append(tags, tag) } request := &elb.AddTagsInput{} request.LoadBalancerNames = []*string{&loadBalancerName} request.Tags = tags _, err := c.elb.AddTags(request) if err != nil { return fmt.Errorf("error adding tags to load balancer: %v", err) } return nil } </ins> // Retrieves instance's vpc id from metadata func (c *Cloud) findVPCID() (string, error) { macs, err := c.metadata.GetMetadata("network/interfaces/macs/")

BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


Question: "//staging/src/k8s.io/apiserver/pkg/storage/etcd3/testing:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/webhook:go_default_library", "//staging/src/k8s.io/client-go/tools/cache:go_default_library", <ins> "//vendor/github.com/go-openapi/spec:go_default_library", </ins> "//vendor/sigs.k8s.io/yaml:go_default_library", ], )

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/google/go-cmp/cmp" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" </ins> "k8s.io/klog/v2/ktesting" <ins> "k8s.io/kubernetes/pkg/features" </ins> clocktesting "k8s.io/utils/clock/testing" "k8s.io/utils/ptr" )

Today the function () only account for the regular containers finish time, presumably assuming that init containers have completed before before. However, with the sidecar (restartable init) containers, sidecar containers will always finish later than regular containers. And those needs to be accounted for the calculation. /kind bug The sidecar finish time will be accounted when calculating the job's finish time. Sidecar container with prolonged termination logic on a job will result in incorrect finish time reporting. /sig apps /priority backlog KEP:
/assign
/triage accepted


Question: if !l.driver.GetDriverInfo().Capabilities[storageframework.CapPVCDataSource] { e2eskipper.Skipf("Driver %q does not support volume clone - skipping", dInfo.Name) } <del> if pattern.VolMode == v1.PersistentVolumeBlock { // TODO: refactor preparePVCDataSourceForProvisioning() below to use // utils.CheckWriteToPath / utils.CheckReadFromPath and remove // redundant InjectContent(). This will enable block volume tests. e2eskipper.Skipf("This test does not support block volumes -- skipping") } </del> // Create a volume expectedContent := fmt.Sprintf("volume content %d", time.Now().UTC().UnixNano())

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --In block volume tests were skipped because is not using , there are more details about how these functions are used in This is a test that uses the , if tested using the it should have , in addition the test cluster should have the Things that are needed for a first time contributor a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if kind can be used a CSI driver like PD CSI Driver, the storage class should be setup to use the xfs fstype the volume snapshot CRDs and controllers installed in the cluster (kubernetes-csi/external-snapshotter#usage) /sig storage /area testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Feature requests are unlikely to make progress as an issue. --/area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### What would you like to be : /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey team! I'd like to further investigate on this issue if possible?

/assign
/area test hey sure, I've more info about how to test it, it requires these components: a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if can be used a CSI driver like , the storage class should be setup to use the fstype the volume snapshot CRDs and controllers installed in the cluster ()
thanks a lot !! I might be a bit slow though, I'm pretty new to the community. I appreciate the documentation you shared, the assignation and how to replicate it, I'll test it out and if I have some doubts, I'll get back with you :D
Hey are you still working on this issue? I am interested in this
Yes, I'm still working on this, but would totally be open to doing pair-programming if you're open to it? I would benefit from you for solving the problem since I'm still pretty new to the codebase and I'm still understanding the environment.
Hey Are you still working on this, I would like to do pair-programming if possible.
Hi there definitely!! Ping me out in slack in the Kubernetes workspace with "Diego Goytia"


Question: <ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Updates the docs to be ready to be used as release docs for a particular # version. # Example usage: # ./versionize-docs.sh v1.0.1 set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. NEW_VERSION=${1-} if [ "$#" -lt 1 ]; then echo "Usage: versionize-docs <release-version>" exit 1 fi SED=sed if which gsed &>/dev/null; then SED=gsed fi if ! ($SED --version 2>&1 | grep -q GNU); then echo "!!! GNU sed is required.  If on OS X, use 'brew install gnu-sed'." exit 1 fi echo "+++ Versioning documentation and examples" # Update the docs to match this version. HTML_PREVIEW_PREFIX="https://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes" md_dirs=(docs examples) md_files=() for dir in "${md_dirs[@]}"; do mdfiles+=($( find "${dir}" -name "*.md" -type f )) done for doc in "${mdfiles[@]}"; do $SED -ri  -e '/<!-- BEGIN STRIP_FOR_RELEASE -->/,/<!-- END STRIP_FOR_RELEASE -->/d'  -e "s|(releases.k8s.io)/[^/]+|1/${NEW_VERSION}|"  "${doc}" # Replace /HEAD in html preview links with /NEW_VERSION. $SED -ri -e "s|(${HTML_PREVIEW_PREFIX}/HEAD)|${HTML_PREVIEW_PREFIX}/${NEW_VERSION}|" "${doc}" is_versioned_tag="<!-- BEGIN MUNGE: IS_VERSIONED --> <!-- TAG IS_VERSIONED --> <!-- END MUNGE: IS_VERSIONED -->" if ! grep -q "${is_versioned_tag}" "${doc}"; then echo -e "nn${is_versioned_tag}nn" >> "${doc}" fi done # Update API descriptions to match this version. $SED -ri -e "s|(releases.k8s.io)/[^/]+|1/${NEW_VERSION}|" pkg/api/v[0-9]*/types.go ${KUBE_ROOT}/hack/run-gendocs.sh ${KUBE_ROOT}/hack/update-swagger-spec.sh </ins>

If you try mimicking the release flow for a release on right now, it's busted: If I generate a dummy commit out of that failure, then run to see what gets generated, it's attempting to actually add the unversioned warnings back in: I was trying to write up a PR to enact something for , but this blocks the release piece for that. Which is fine, we have some time.
cc
Running gendocs should add the unversioned warnings, unless the doc has <!-- TAG IS_VERSIONED --. Do you know if mark-new-version that tag before failing?
Ah, I think I see what might be happening. This is the post- version of Notice the newline fun.
Sorry which newline are you talking about? :)
My PR fixes that
Ah, yes, cool. It would help if you flagged functional changes along more loudly. :)
Yea, sorry about that. Have updated the PR description to highlight each fix.


Question: # Post-condition: nginx0 & nginx1 should both have paused set to nothing, and since nginx2 is malformed, it should error kube::test::get_object_assert deployment "{{range.items}}{{.spec.paused}}:{{end}}" "<no value>:<no value>:" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" <ins> ## Fetch rollout status for multiple resources output_message=$(! kubectl rollout status -f hack/testdata/recursive/deployment/deployment --timeout=1s 2>&1 "${kube_flags[@]:?}") # Post-condition: nginx1 should both exist and nginx2 should error kube::test::if_has_string "${output_message}" "Waiting for deployment "nginx1-deployment" rollout to finish" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" ## Fetch rollout status for deployments recursively output_message=$(! kubectl rollout status -f hack/testdata/recursive/deployment -R --timeout=1s 2>&1 "${kube_flags[@]:?}") # Post-condition: nginx0 & nginx1 should both exist, nginx2 should error kube::test::if_has_string "${output_message}" "Waiting for deployment "nginx0-deployment" rollout to finish" kube::test::if_has_string "${output_message}" "Waiting for deployment "nginx1-deployment" rollout to finish" kube::test::if_has_string "${output_message}" "Object 'Kind' is missing" </ins> ## Retrieve the rollout history of the deployments recursively output_message=$(! kubectl rollout history -f hack/testdata/recursive/deployment --recursive 2>&1 "${kube_flags[@]}") # Post-condition: nginx0 & nginx1 should both have a history, and since nginx2 is malformed, it should error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: fails with . What you expected to happen: The suggest that should work for multiple resources in a directory when running . should've been applied to all resources under following was omitted in and I wasn't able to find a discussion regarding this omission in other issues. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten I agree with the assessment that this is a bug the , , and all work with spec files with multiple resources in them. does not. All of the commands would additionally benefit from only operating on supported resource types for rollout (Deployment, DaemonSet) from the spec file, but that's really a sperate enhancement. Client kubectl 1.16.6 Server EKS 1.14.9 How Status Behaves: How Pause Behaves: How Resume Behaves: How History Behaves:
You can't reopen an issue/PR unless you authored it or you are a collaborator.

<detailsIn response to : /reopen /remove-lifecycle rotten Server EKS 1.14.9 kubectl rollout status -f .ClusterServicecore- -n kube-system error: rollout status is only supported on individual resources and resource collections - 6 resources were found kubectl rollout pause -f .ClusterServicecore- -n kube-system paused error: serviceaccounts "coredns" pausing is not supported error: "system:coredns" pausing is not supported error: "system:coredns" pausing is not supported error: configmaps "coredns" pausing is not supported error: services "kube-dns" pausing is not supported kubectl rollout resume -f .dc-infra-k8s-infrastructureClusterServicecore- -n kube-system resumed error: serviceaccounts "coredns" resuming is not supported error: "system:coredns" resuming is not supported error: "system:coredns" resuming is not supported error: configmaps "coredns" resuming is not supported error: services "kube-dns" resuming is not supported Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you reopen this? If not I guess I'll file a new issue.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What is needed to move this forward?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle stale poke poke
There appears to be a PR that was closed without merging Strangely no changes are recorded in the PR
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Any updates on this?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "scheme": "HTTPS", "host": "127.0.0.1", "port": {{secure_port}}, <del> "path": "/healthz" </del> <ins> "path": "/readyz" </ins> }, "periodSeconds": 1, "timeoutSeconds": 15

Add readyz to GCE manifest for readiness checking on the apiserver. /assign
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <ins> #!/usr/bin/env bash # Copyright 2020 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail run_exec_credentials_tests() { set -o nounset set -o errexit kube::log::status "Testing kubectl with configured exec credentials plugin" cat > "${TMPDIR:-/tmp}"/invalid_exec_plugin.yaml << EOF apiVersion: v1 clusters: - cluster: name: test contexts: - context: cluster: test user: invalid_token_user name: test current-context: test kind: Config preferences: {} users: - name: invalid_token_user user: exec: apiVersion: client.authentication.k8s.io/v1beta1 # Any invalid exec credential plugin will do to demonstrate command: ls EOF ### Provided --token should take precedence, thus not triggering the (invalid) exec credential plugin # Pre-condition: Client certificate authentication enabled on the API server kube::util::test_client_certificate_authentication_enabled # Command output=$(kubectl "${kube_flags_with_token[@]:?}" --kubeconfig="${TMPDIR:-/tmp}"/invalid_exec_plugin.yaml get namespace kube-system -o name || true) if [[ "${output}" == "namespace/kube-system" ]]; then kube::log::status "exec credential plugin not triggered since kubectl was called with provided --token" else kube::log::status "Unexpected output when providing --token for authentication - exec credential plugin likely triggered. Output: ${output}" exit 1 fi # Post-condition: None ### Without provided --token, the exec credential plugin should be triggered # Pre-condition: Client certificate authentication enabled on the API server - already checked by positive test above # Command output2=$(kubectl "${kube_flags_without_token[@]:?}" --kubeconfig="${TMPDIR:-/tmp}"/invalid_exec_plugin.yaml get namespace kube-system -o name 2>&1 || true) if [[ "${output2}" =~ "json parse error" ]]; then kube::log::status "exec credential plugin triggered since kubectl was called without provided --token" else kube::log::status "Unexpected output when not providing --token for authentication - exec credential plugin not triggered. Output: ${output2}" exit 1 fi # Post-condition: None rm "${TMPDIR:-/tmp}"/invalid_exec_plugin.yaml set +o nounset set +o errexit } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval.

Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called.

third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: touch "${KUBE_ROOT}/staging/src/${repo}/BUILD" done <ins> # Warning: Please be careful when ignoring these pkg-config errors. # As gazelle doesn't support pkg-config, if we suppress these errors here, we actually need to modify the script below to handle it manually. # For instance, in the case of `libseccomp`, we set the link option `-lseccomp` to make it work properly. KNOWN_PKG_CONFIG_ERRORS=( "vendor/github.com/seccomp/libseccomp-golang/seccomp(_internal)?.go: pkg-config not supported: #cgo pkg-config: libseccomp" "vendor/github.com/google/cadvisor/nvm/machine_libipmctl.go: pkg-config not supported: #cgo pkg-config: libipmctl" ) </ins> # Run gazelle to update Go rules in BUILD files. # filter out known pkg-config error (see buildozer workaround below) # NOTE: the `|| exit "${PIPESTATUS[0]}"` is to ignore grep errors

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Got the following error while trying to run : How to reproduce it (as minimally and precisely as possible): Ran this on a fresh install of ubuntu 19.04, Go 1.12.6, and bazel 0.26.1. Anything else we need to know?: Seems like the change might have been due to the update in vendored deps from ( ). Although this script is run in CI where it is working so may be some set of recommende pkg versions would help prevent this problem? Environment: Cloud provider or hardware configuration: OS (e.g: ): ubuntu 19.04 Kernel (e.g. ): 5.0.0-15-generic bazel: 0.26.1 Go: 1.12.6
Not sure who owns this... /sig testing
Seems like the errors are "warnings" and can be ignored? It seems that BUILD files are properly updated by update-bazel, And bazel builds do work properly... /cc
/sig release this is not a failure /retitle hack/update-bazel produces warnings there was some discussion on the original PR about filtering them out but nothing conclusive I'll file a PR to do that now.. /assign
aside: this is actually from :-) we're working around the lack of support because we only need it for this package and on all known systems we compile on so far it only needs the one link flag as previously specified in the cgo config.
filed
Glad to see it. The output message is so misleading that I have spent hours on this issue
sorry about that the output does say "error" but it's actually considered a non-fatal warning in the upstream code and doesn't cause the script (or build) to fail.
was merged before this discussion was resolved , so we kinda saw this coming. :/ Hopefully we can get the fix from in merged as quick as possible
/reopen we have a new a new occurrence with libipmctl
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have the same question. This output is a bit annoying.
We can update the script to handle this issue and suppress the warning. The previous PR is an example. On Thu, Apr 16, 2020, 12:14 AM <:
/assign


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package set import ( "bytes" "fmt" "io" "io/ioutil" "net/http" "path" "testing" "github.com/stretchr/testify/assert" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/rest/fake" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/testapi" "k8s.io/kubernetes/pkg/apis/apps" "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/pkg/apis/extensions" cmdtesting "k8s.io/kubernetes/pkg/kubectl/cmd/testing" "k8s.io/kubernetes/pkg/kubectl/resource" "k8s.io/kubernetes/pkg/printers" ) const serviceAccount = "serviceaccount1" const serviceAccountMissingErrString = "serviceaccount is required" const resourceMissingErrString = `You must provide one or more resources by argument or filename. Example resource specifications include: '-f rsrc.yaml' '--filename=rsrc.json' '<resource> <name>' '<resource>'` func TestServiceAccountLocal(t *testing.T) { inputs := []struct { yaml     string apiGroup string }{ {yaml: "../../../../test/fixtures/doc-yaml/user-guide/replication.yaml", apiGroup: api.GroupName}, {yaml: "../../../../test/fixtures/doc-yaml/admin/daemon.yaml", apiGroup: extensions.GroupName}, {yaml: "../../../../test/fixtures/doc-yaml/user-guide/replicaset/redis-slave.yaml", apiGroup: extensions.GroupName}, {yaml: "../../../../test/fixtures/doc-yaml/user-guide/job.yaml", apiGroup: batch.GroupName}, {yaml: "../../../../test/fixtures/doc-yaml/user-guide/deployment.yaml", apiGroup: extensions.GroupName}, {yaml: "../../../../examples/storage/minio/minio-distributed-statefulset.yaml", apiGroup: apps.GroupName}, } f, tf, _, _ := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ APIRegistry: api.Registry, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" out := new(bytes.Buffer) cmd := NewCmdServiceAccount(f, out, out) cmd.SetOutput(out) cmd.Flags().Set("output", "yaml") cmd.Flags().Set("local", "true") for _, input := range inputs { testapi.Default = testapi.Groups[input.apiGroup] tf.Printer = printers.NewVersionedPrinter(&printers.YAMLPrinter{}, testapi.Default.Converter(), *testapi.Default.GroupVersion()) saConfig := serviceAccountConfig{fileNameOptions: resource.FilenameOptions{ Filenames: []string{input.yaml}}, out:   out, local: true} err := saConfig.Complete(f, cmd, []string{serviceAccount}) assert.NoError(t, err) err = saConfig.Run() assert.NoError(t, err) assert.Contains(t, out.String(), "serviceAccountName: "+serviceAccount, fmt.Sprintf("serviceaccount not updated for %s", input.yaml)) } } func TestServiceAccountRemote(t *testing.T) { inputs := []struct { object              runtime.Object apiPrefix, apiGroup string args                []string }{ { object: &extensions.ReplicaSet{ TypeMeta:   metav1.TypeMeta{Kind: "ReplicaSet", APIVersion: api.Registry.GroupOrDie(extensions.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/apis", apiGroup: extensions.GroupName, args: []string{"replicaset", "nginx", serviceAccount}, }, { object: &extensions.DaemonSet{ TypeMeta:   metav1.TypeMeta{Kind: "DaemonSet", APIVersion: api.Registry.GroupOrDie(extensions.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/apis", apiGroup: extensions.GroupName, args: []string{"daemonset", "nginx", serviceAccount}, }, { object: &api.ReplicationController{ TypeMeta:   metav1.TypeMeta{Kind: "ReplicationController", APIVersion: api.Registry.GroupOrDie(api.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/api", apiGroup: api.GroupName, args: []string{"replicationcontroller", "nginx", serviceAccount}}, { object: &extensions.Deployment{ TypeMeta:   metav1.TypeMeta{Kind: "Deployment", APIVersion: api.Registry.GroupOrDie(extensions.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/apis", apiGroup: extensions.GroupName, args: []string{"deployment", "nginx", serviceAccount}, }, { object: &batch.Job{ TypeMeta:   metav1.TypeMeta{Kind: "Job", APIVersion: api.Registry.GroupOrDie(batch.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/apis", apiGroup: batch.GroupName, args: []string{"job", "nginx", serviceAccount}, }, { object: &apps.StatefulSet{ TypeMeta:   metav1.TypeMeta{Kind: "StatefulSet", APIVersion: api.Registry.GroupOrDie(apps.GroupName).GroupVersion.String()}, ObjectMeta: metav1.ObjectMeta{Name: "nginx"}, }, apiPrefix: "/apis", apiGroup: apps.GroupName, args: []string{"statefulset", "nginx", serviceAccount}, }, } for _, input := range inputs { groupVersion := api.Registry.GroupOrDie(input.apiGroup).GroupVersion testapi.Default = testapi.Groups[input.apiGroup] f, tf, codec, _ := cmdtesting.NewAPIFactory() tf.Printer = printers.NewVersionedPrinter(&printers.YAMLPrinter{}, testapi.Default.Converter(), *testapi.Default.GroupVersion()) tf.Namespace = "test" tf.CategoryExpander = resource.LegacyCategoryExpander tf.Client = &fake.RESTClient{ APIRegistry:          api.Registry, NegotiatedSerializer: testapi.Default.NegotiatedSerializer(), Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { resourcePath := testapi.Default.ResourcePath(input.args[0]+"s", tf.Namespace, input.args[1]) switch p, m := req.URL.Path, req.Method; { case p == resourcePath && m == http.MethodGet: return &http.Response{StatusCode: http.StatusOK, Header: defaultHeader(), Body: objBody(codec, input.object)}, nil case p == resourcePath && m == http.MethodPatch: stream, err := req.GetBody() if err != nil { return nil, err } bytes, err := ioutil.ReadAll(stream) if err != nil { return nil, err } assert.Contains(t, string(bytes), `"serviceAccountName":`+`"`+serviceAccount+`"`, fmt.Sprintf("serviceaccount not updated for %#v", input.object)) return &http.Response{StatusCode: http.StatusOK, Header: defaultHeader(), Body: objBody(codec, input.object)}, nil default: t.Errorf("%s: unexpected request: %s %#vn%#v", "serviceaccount", req.Method, req.URL, req) return nil, fmt.Errorf("unexpected request") } }), VersionedAPIPath: path.Join(input.apiPrefix, groupVersion.String()), GroupName:        input.apiGroup, } out := new(bytes.Buffer) cmd := NewCmdServiceAccount(f, out, out) cmd.SetOutput(out) cmd.Flags().Set("output", "yaml") saConfig := serviceAccountConfig{ out:   out, local: false} err := saConfig.Complete(f, cmd, input.args) assert.NoError(t, err) err = saConfig.Run() assert.NoError(t, err) } } func TestServiceAccountValidation(t *testing.T) { inputs := []struct { args        []string errorString string }{ {args: []string{}, errorString: serviceAccountMissingErrString}, {args: []string{serviceAccount}, errorString: resourceMissingErrString}, } for _, input := range inputs { f, tf, _, _ := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ APIRegistry: api.Registry, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" out := bytes.NewBuffer([]byte{}) cmd := NewCmdServiceAccount(f, out, out) cmd.SetOutput(out) saConfig := &serviceAccountConfig{} err := saConfig.Complete(f, cmd, input.args) assert.EqualError(t, err, input.errorString) } } func objBody(codec runtime.Codec, obj runtime.Object) io.ReadCloser { return bytesBody([]byte(runtime.EncodeOrDie(codec, obj))) } func defaultHeader() http.Header { header := http.Header{} header.Set("Content-Type", runtime.ContentTypeJSON) return header } func bytesBody(bodyBytes []byte) io.ReadCloser { return ioutil.NopCloser(bytes.NewReader(bodyBytes)) } </ins>

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: k8s.io/apiserver/pkg/server/options k8s.io/apiserver/pkg/server/options/encryptionconfig k8s.io/apiserver/pkg/server/options/encryptionconfig/controller <ins> k8s.io/apiserver/pkg/server/options/encryptionconfig/metrics </ins> k8s.io/apiserver/pkg/server/resourceconfig k8s.io/apiserver/pkg/server/routes k8s.io/apiserver/pkg/server/storage

ref - /sig auth
/triage accepted
/retitle encryption-at-rest: add success or failure metric when encryption configuration attempts to dynamically update
/assign


Question: ## Requirements <del> To run Kubemark you need a Kubernetes cluster for running all your HollowNodes and a dedicated machine for a master. Master machine has to be directly routable from HollowNodes. You also need an access to some Docker repository. </del> <ins> To run Kubemark you need a Kubernetes cluster (called `external cluster`) for running all your HollowNodes and a dedicated machine for a master. Master machine has to be directly routable from HollowNodes. You also need an access to some Docker repository. </ins> Currently scripts are written to be easily usable by GCE, but it should be relatively straightforward to port them to different providers or bare metal.

A glitch I experienced when following the : Although it might be obvious, it doesn't hurt to expand "external cluster" to "external kubernetes cluster". Also, the guide doesn't mention I need to configure my kubectl to target the "external kubernetes cluster" while it's actually required.


Question: Help:           "Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ.", StabilityLevel: metrics.ALPHA, }, []string{"queue"}) <ins> SchedulerGoroutines = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      SchedulerSubsystem, Name:           "scheduler_goroutines", Help:           "Number of running goroutines split by the work they do such as binding.", StabilityLevel: metrics.ALPHA, }, []string{"work"}) </ins> PodSchedulingDuration = metrics.NewHistogram( &metrics.HistogramOpts{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


Question: want:      list(11, "rv:20"), wantPaged: true, }, <ins> { name:      "two pages with resourceVersion and resourceVersionMatch", fields:    fields{PageSize: 10, PageFn: (&testPager{t: t, expectPage: 10, remaining: 11, rv: "rv:20"}).PagedList}, args:      args{options: metav1.ListOptions{ResourceVersion: "rv:10", ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan}}, want:      list(11, "rv:20"), wantPaged: true, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

We recently started using the list pager from client-go as part of Calico, and have seen reports of the following error showing up: No error, just beautiful paged list calls. Create 1,000 resources of a particular type, such as a NetworkPolicy. (Need 500 to trigger paging) Do a of all those resources with set. Originally filed here (but maybe that repo is not monitored): Reference issue in Calico: projectcalico/calico (comment) Reference Calico list pager code: We think this is the required fix: <detailsclient-go version (but bug seems to be in master too). </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" utilerrors "k8s.io/apimachinery/pkg/util/errors" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apiserver/pkg/admission" <ins> "k8s.io/klog/v2" </ins> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/core/pods" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We enabled AlwaysPullImages admission in an existing cluster, and found annotations/labels/finalizers of existing Pods whose imagePullPolicy configured as could not be updated. AlwaysPullImages admission mutated containerSpec by . And this change later got denied by Pod validation. It also caused terminating Pods with finalizers stuck forever until we removed AlwaysPullImages admission. What you expected to happen: Existing Pods are not affected by AlwaysPullImages admission controller. How to reproduce it (as minimally and precisely as possible): the kube-apiserver with AlwaysPullImages admission controller disabled a Pod with imagePullPolicy configured as in its containers AlwaysPullImages admission controller annotation to Pod Anything else we need to know?: If this is confirmed a bug, I can submit a PR to fix this. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
seems to be a bug. /assign
/remove-sig api-machinery /sig node auth
try to fix it in When AlwaysPullImages is disabled, you cannot update imagePullPolicy of pod. The error message is the same as yours. imagePullPolicy is immutable. When AlwaysPullImages is enabled, old pod(with imagePullPolicy=ifNotPresent) update will change imagePullPolicy as well. Workaround: recreated pod -_-
Thanks for the fixes. I still have regarding the PR. IMHO, it's OK that imagePullPolicy is not updatable. Why don't we just ignore Update in the admission, but choose to update spec instead? Any concerns?

This is another fix option. I'm not sure which is better. The pod will be still ifNotPresent after updating, even if the AlwaysPullImages is enabled. Which is expected behavior?
In my case, I would expect the admission not to touch existing Pods. Just want to make sure if there's any concerns.
I would recommend ignoring pod updates in AlwaysPullImages if the update does not change the set of image names referenced by containers in the pod. Changing images inline in a pod update is unusual, and that change would allow unrelated updates (to annotations, labels, etc) to proceed
Agree.
is updated .
This issue is fixed by


Question: Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, <del> "containerB": stoppedState, </del> <ins> "containerB": succeededState, }, Host: "machine", }, }, api.PodRunning, "mixed state #1 with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, }, Host: "machine", }, }, api.PodPending, "mixed state #2 with restart never", }, } for _, test := range tests { if status, err := getPodStatus(test.pod, fakeClient.Nodes()); status != test.status { t.Errorf("In test %s, expected %v, got %v", test.test, test.status, status) if err != nil { t.Errorf("In test %s, unexpected error: %v", test.test, err) } } } } func TestPodStatusWithRestartOnFailure(t *testing.T) { fakeClient := client.Fake{ MinionsList: api.NodeList{ Items: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "machine"}, }, }, }, } desiredState := api.PodSpec{ Containers: []api.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: api.RestartPolicy{OnFailure: &api.RestartPolicyOnFailure{}}, } currentState := api.PodStatus{ Host: "machine", } runningState := api.ContainerStatus{ State: api.ContainerState{ Running: &api.ContainerStateRunning{}, }, } succeededState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: 0, }, }, } failedState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: -1, }, }, } tests := []struct { pod    *api.Pod status api.PodPhase test   string }{ {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": runningState, }, Host: "machine", }, }, api.PodRunning, "all running with restart onfailure", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": succeededState, "containerB": succeededState, }, Host: "machine", }, }, api.PodSucceeded, "all succeeded with restart onfailure", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": failedState, "containerB": failedState, }, Host: "machine", }, }, api.PodRunning, "all failed with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": succeededState, </ins> }, Host: "machine", }, }, api.PodRunning, <del> "mixed state #1", </del> <ins> "mixed state #1 with restart onfailure", </ins> }, { &api.Pod{

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: } var resolver *claimResolver <del> if opts.JWTAuthenticator.ClaimMappings.Groups.Claim != "" { resolver = newClaimResolver(opts.JWTAuthenticator.ClaimMappings.Groups.Claim, client, verifierConfig) </del> <ins> groupsClaim := opts.JWTAuthenticator.ClaimMappings.Groups.Claim if groupsClaim != "" { resolver = newClaimResolver(groupsClaim, client, verifierConfig) </ins> } authenticator := &Authenticator{

Follow up PR: would be good to use some local vars to make these statements shorter. Originally posted by in
/sig auth /triage accepted /kind cleanup /assign


Question: // 30 seconds by default. // Based on the above check if the pod receives the traffic. ginkgo.By("checking client pod connected to the backend 1 on Node IP " + serverNodeInfo.nodeIP) <del> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend1)); err != nil { </del> <ins> if err := wait.PollImmediate(5*time.Second, time.Minute, logContainsFn(podBackend1, podClient)); err != nil { </ins> logs, err = e2epod.GetPodLogs(cs, ns, podClient, podClient) framework.ExpectNoError(err) framework.Logf("Pod client logs: %s", logs)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale