Question: filter parent 1: protocol ip pref 1 u32 fh 800::801 order 2049 key ht 800 bkt 0 flowid 1:2 match 01020000/ffff0000 at 16 ` <ins> var tcFilterOutputNewVersion = `filter parent 1: protocol ip pref 1 u32 filter parent 1: protocol ip pref 1 u32 chain 0 fh 800: ht divisor 1 filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 not_in_hw match ac110002/ffffffff at 16 filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::801 order 2049 key ht 800 bkt 0 flowid 1:2 not_in_hw match 01020000/ffff0000 at 16 ` </ins> func TestFindCIDRClass(t *testing.T) { tests := []struct {

What happened: I tried to set up bandwidth limiting by booting a cluster with , which by default sets up kubenet. I used the annotations xM xM and then used to test the ingress and egress bandwidths, with no effect. Does anyone know if this feature is still supported in kubenet? And if so what are the steps I need to take to boot a cluster that can use pod annotations to do tc shaping. What you expected to happen: Per traffic shaping should be implemented on Kubenet as of 4/12/2018. Anything else we need to know?: There are a lot of issues regarding traffic shaping not working. None have a concrete answer of how to get it to work on kubenet: Relevant: I believe this is where it was first implemented. Environment: Kubernetes version (use ): 1.11.8 Cloud provider or hardware configuration: AWS CC:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/triage unresolved I am a bot run by
can we put this on sig-network agenda? the discussion item would be: what is the current status of this feature (alpha, beta, GA?) what should we do next with this API?
I have this to the agenda, will grab some info.
AFAIK traffic shaping is still supported in kubenet. Does a pod with these annotations come up successfully but has no traffic shaping applied? Or do things fail earlier? If it comes up successfully, could you run ?
On GKE we do not officially support this, our default image even don't have tc installed. I tried with ubuntu, the kubenet is failed to parse the tc output. kubenet expect. // filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 tc return.// filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 notinhw
If this issue has been triaged, please comment .

Meta: /athenabot mark-triage-reminder I am a bot run by
Depending on which CNI plugin you use you need to install the bandwidth plugin, see:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
since it is in progress /priority critical-urgent /milestone v1.17
/lifecycle frozen
/assign
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
The fix is still pending in review.
Will it be merged by this Thursday for v1.17 code freeze?
/milestone v1.18 (v1.17 code freeze)
matching assigned priority on PR
Hello I'm one of the Bug Triage shadows for the 1.18 release. Since this issue is tagged for 1.18, I'd like to check its status. Is this issue intended for milestone 1.18? Friendly reminder: Code freeze is scheduled for March 5th.
Hello Checking in again. Do you think this will make it in for 1.18 release?
Hey Checking in again as this issue is marked as . Are you close to completing this for v1.18 release? Another reminder that code freeze is just around the corner! (March 5th).
Friendly reminder that code freeze is in 2 days and this issue is marked as
The fix got approved by ,should be able to catch the freeze.


Question: "errors" "fmt" "reflect" <ins> "strconv" </ins> "sync" "testing" "time"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key.

What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring.

The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.


Question: return out } <ins> func filterOut(ss []string, x string) ([]string, bool) { var filtered []string for i, s := range ss { if s == x { if filtered == nil { filtered = make([]string, i, len(ss)) copy(filtered, ss[:i]) } } else if filtered != nil { filtered = append(filtered, s) } } if filtered != nil { if len(filtered) == 0 { return nil, true } return filtered, true } return ss, false } </ins>

When doing in an OpenShift cluster with a nullable field being required (compare ) and of value null, one gets:
cc
is there a way to also fix the faulty kubectl validation in 1.18.x?


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v2beta1 import ( "testing" "github.com/stretchr/testify/assert" "k8s.io/api/autoscaling/v2beta1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/kubernetes/pkg/apis/autoscaling" ) // Testing nil pointer panic uncovered by #70806 // TODO(yue9944882): Test nil/empty conversion across all resource types func TestNilOrEmptyConversion(t *testing.T) { scheme := runtime.NewScheme() assert.NoError(t, addConversionFuncs(scheme)) testCases := []struct { obj1 interface{} obj2 interface{} }{ { obj1: &autoscaling.ExternalMetricSource{}, obj2: &v2beta1.ExternalMetricSource{}, }, { obj1: &autoscaling.ExternalMetricStatus{}, obj2: &v2beta1.ExternalMetricStatus{}, }, { obj1: &autoscaling.PodsMetricSource{}, obj2: &v2beta1.PodsMetricSource{}, }, { obj1: &autoscaling.PodsMetricStatus{}, obj2: &v2beta1.PodsMetricStatus{}, }, { obj1: &autoscaling.ObjectMetricSource{}, obj2: &v2beta1.ObjectMetricSource{}, }, { obj1: &autoscaling.ObjectMetricStatus{}, obj2: &v2beta1.ObjectMetricStatus{}, }, { obj1: &autoscaling.ResourceMetricSource{}, obj2: &v2beta1.ResourceMetricSource{}, }, { obj1: &autoscaling.ResourceMetricStatus{}, obj2: &v2beta1.ResourceMetricStatus{}, }, { obj1: &autoscaling.HorizontalPodAutoscaler{}, obj2: &v2beta1.HorizontalPodAutoscaler{}, }, { obj1: &autoscaling.MetricTarget{}, obj2: &v2beta1.CrossVersionObjectReference{}, }, } for _, testCase := range testCases { assert.NoError(t, scheme.Convert(testCase.obj1, testCase.obj2, nil)) assert.NoError(t, scheme.Convert(testCase.obj2, testCase.obj1, nil)) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g.

): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all.

I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix?

:)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via


Question: // SkipUnlessMasterOSDistroIs skips if the master OS distro is not included in the supportedMasterOsDistros. func SkipUnlessMasterOSDistroIs(supportedMasterOsDistros ...string) { if !framework.MasterOSDistroIs(supportedMasterOsDistros...) { <del> skipInternalf(1, "Only supported for master OS distro %v (not %s)", supportedMasterOsDistros, TestContext.MasterOSDistro) </del> <ins> skipInternalf(1, "Only supported for master OS distro %v (not %s)", supportedMasterOsDistros, framework.TestContext.MasterOSDistro) </ins> } } // SkipUnlessNodeOSDistroIs skips if the node OS distro is not included in the supportedNodeOsDistros. func SkipUnlessNodeOSDistroIs(supportedNodeOsDistros ...string) { if !framework.NodeOSDistroIs(supportedNodeOsDistros...) { <del> skipInternalf(1, "Only supported for node OS distro %v (not %s)", supportedNodeOsDistros, TestContext.NodeOSDistro) </del> <ins> skipInternalf(1, "Only supported for node OS distro %v (not %s)", supportedNodeOsDistros, framework.TestContext.NodeOSDistro) </ins> } } // SkipIfNodeOSDistroIs skips if the node OS distro is included in the unsupportedNodeOsDistros. func SkipIfNodeOSDistroIs(unsupportedNodeOsDistros ...string) { if framework.NodeOSDistroIs(unsupportedNodeOsDistros...) { <del> skipInternalf(1, "Not supported for node OS distro %v (is %s)", unsupportedNodeOsDistros, TestContext.NodeOSDistro) </del> <ins> skipInternalf(1, "Not supported for node OS distro %v (is %s)", unsupportedNodeOsDistros, framework.TestContext.NodeOSDistro) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. They are skipped because they require SSH access to drop the Flexvolume plugin onto the host, but these test clusters do not have SSH access. We should change all Flexvolume tests to use the DaemonSet installation mechanism instead which does not require SSH access. See and Flexvolume tests I'm aware of are below, but there could be more: Looking for help with this bug! /sig storage /priority important-soon /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. /priority important-soon /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Can I work on this issue? I think i know how to handle it.
are you still working on this? I'm interested in helping out as well.
/assign


Question: return err } <del> podName := volumetypes.UniquePodName("") </del> <ins> podName := nestedpendingoperations.EmptyUniquePodName </ins> // TODO: remove this -- not necessary if !volumeToMount.PluginIsAttachable { // Non-attachable volume plugins can execute mount for multiple pods

During testing a AWS cluster, found AWS volume once was pending for detach about 30 minutes. According to the log, the following happened, controller issues a detach operation after detach response returns, it periodically checks the status until it reaches "detached". However, in this test, the status is "busy" all the time. after reaching the timeout 30 mins, detach function returns error because the status is not "detached" The following is the log related to the failed test I0915 18:29:46. 5 ] Verified volume is safe to detach for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-". E0915 18:59:54. 5 ] Error detaching volumeID "vol-": Timeout waiting for volume state: actual=busy, desired=detached E0915 18:59:54. 5 ] Operation for """" failed. No retries permitted until 2016-09-15 18:59:55. +0000 UTC (durationBeforeRetry 500ms). Error: DetachVolume.Detach failed for volume "" (spec.Name: "pvc--7ac7-11e6-abe5-") from node "ip-172-20-0-" with: Timeout waiting for volume state: actual=busy, desired=detached
Thoughts on what the ideal timeout should be. Too short would be bad because it would give a false-negative.
I know this is from a test but I'm seeing the same problem on a production cluster (1.4.8). I'm not sure exactly what triggers it yet. Kubelet only unmounts the pod directory, not the mount in "aws-ebs/mounts". As a result, the EBS gets stuck in "busy" state when detaching since it is still mounted to the VM. Manually unmounting the EBS allows it to get unstuck and detach safely. In controller manager I see: It then proceeds to get stuck detaching (volume goes to busy state), after which it starts trying to force detach which of course hangs indefinitely.

Inspecting the node directly, I see it's still mounted which is why detach hangs: So something in kubelet is messed up and not unmounting both mount points correctly.
I think this might be the same issue as
On kubelet, it looks like the volume failed to mark as mounted, which is why it subsequently got stuck. This may have to do with the fact this pod () got rolled back while waiting for the previous deployment to succeed.
^ Looks like it might be a race condition between the controller manager and reconciler: manager attaches volume starts attachment process (pod rollback/delete) controller manager thinks its safe to detach, so removes volume from node's volume list, then tries to detach mounts the volume as part of the ongoing attach/mount process from (2) tries to update the mount list, this fails because something (reconciler?) has removed the volume from attached list gets stuck in (busy) state detaching, as its mounted to the kubelet node. it never gets unmounted again in kubelet because state doesn't reflect it is mounted.
Looks like there exists race conditions between external updates of and 's asynchronous kick off of operations like MountVolume, which is the case I'm seeing. Seems like a lot of these edge cases could crop up here - maybe reconciler should lock changes to the desiredStateOfWorld while it's running?
thank you for providing detailed information and analysis. I think there is a race condition between asynchronous operation MountVolume and updating actualStateOfWorld directly by reconciler I will work on a fix. Please let me know if you have any issue/comments.


Question: if pvcSpec.DataSource.APIGroup != nil { apiGroup = *pvcSpec.DataSource.APIGroup } <del> if utilfeature.DefaultFeatureGate.Enabled(features.VolumePVCDataSource) && pvcSpec.DataSource.Kind == pvc && </del> <ins> if pvcSpec.DataSource.Kind == pvc && </ins> apiGroup == "" { return true

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//vendor/k8s.io/api/authentication/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/clock:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", <del> "//vendor/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library", </del> <ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", </ins> ], )

/kind bug /priority failing-test /priority important-critical /milestone v1.11 /assign cc All GKE jobs are failing in various release dashboards. Code freeze for 1.11 is a couple of hours away.
Common error appears to be:
Failures seem to align with when was merged. But nothing in that PR seems suspicious. Having verify. Bringing up my own cluster on head to try and get a live cluster.
I have a quick try to reproduce one of the failure test cases by gcloud container clusters create --addons=HttpLoadBalancing,HorizontalPodAutoscaling,KubernetesDashboard --machine-type=n1-standard-2 --image-type=gci --num-nodes=3 --cluster-version=1.11.0-beta.1.222+ test- The cluster isn't up. What I observed is the following: 1) kubelet on master is in crash-loop due to a panic No other master components could be up. 2) kubelet is in crashloop in node as well.
/sig node Looks like has a patch up at
/priority critical-urgent
/status approved-for-milestone
Waiting for new runs to verify the fix. Will reopen this issue if needed
/reopen Reopening this one until its actually verified to ensure it shows up in our tracking queries.
I see the GKE tests cleaning up in the . still doesn't seem to have the change (will have to wait for cutover?). The fix looks promising. Will wait for a couple more runs before I close
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Action Required: This issue has not been updated since Jun 6. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
all the GKE ones still look good /close


Question: Host: "machine", }, }, <del> api.PodFailed, "all stopped", </del> <ins> api.PodRunning, "all stopped with restart always", </ins> }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ <del> "containerA": stoppedState, </del> <ins> "containerA": runningState, </ins> "containerB": stoppedState, }, <del> Host: "machine-two", </del> <ins> Host: "machine", }, }, api.PodRunning, "mixed state #1 with restart always", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, }, Host: "machine", }, }, api.PodPending, "mixed state #2 with restart always", }, } for _, test := range tests { if status, err := getPodStatus(test.pod, fakeClient.Nodes()); status != test.status { t.Errorf("In test %s, expected %v, got %v", test.test, test.status, status) if err != nil { t.Errorf("In test %s, unexpected error: %v", test.test, err) } } } } func TestPodStatusWithRestartNever(t *testing.T) { fakeClient := client.Fake{ MinionsList: api.NodeList{ Items: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "machine"}, }, }, }, } desiredState := api.PodSpec{ Containers: []api.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: api.RestartPolicy{Never: &api.RestartPolicyNever{}}, } currentState := api.PodStatus{ Host: "machine", } runningState := api.ContainerStatus{ State: api.ContainerState{ Running: &api.ContainerStateRunning{}, }, } succeededState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: 0, }, }, } failedState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: -1, }, }, } tests := []struct { pod    *api.Pod status api.PodPhase test   string }{ {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": runningState, }, Host: "machine", }, }, api.PodRunning, "all running with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": succeededState, "containerB": succeededState, }, Host: "machine", }, }, api.PodSucceeded, "all succeeded with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": failedState, "containerB": failedState, }, Host: "machine", </ins> }, }, api.PodFailed, <del> "all stopped but minion missing", </del> <ins> "all failed with restart never", </ins> }, { &api.Pod{

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: } return podNames, nil } <ins> // WaitForPodContainerToFail waits for the given Pod container to fail with the given reason, specifically due to // invalid container configuration. In this case, the container will remain in a waiting state with a specific // reason set, which should match the given reason. func WaitForPodContainerToFail(c clientset.Interface, namespace, podName string, containerIndex int, reason string, timeout time.Duration) error { return wait.PollImmediate(poll, timeout, podContainerFailed(c, namespace, podName, containerIndex, reason)) } </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: N/A Which test(s) are failing: should fail substituting values in a volume subpath with absolute path Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Some of the tests are negative test cases which are supposed to ensure that those invalid usecases are handled properly. However, some of the tests are false positives, they can pass due to various reasons. One such example is: should fail substituting values in a volume subpath with absolute path. This test can pass if: the Pod cannot start due to various reasons (e.g.: the container image cannot be pulled or does not exist). the Pod ran to completion, even though the container was not supposed to start in the first place. Anything else we need to know: /sig testing Related:
/sig storage SIG storage owns the test case, though I see that you are changing core framework methods to fix this, so leaving SIG testing. Looks like also possibly sig-node, skimming the linked PRs Based on the PRs this also appears to be related to windows? (unsure, no mention here)
I've Related: to the issue description. After fixing this test, I've noticed that it now fails on Windows, which means the issue was masked by the test's false positive result. Basically, it means that you can have volume subpaths with absolute paths substituted on variable expansion and have Windows containers use them and be running, which according to the test, it shouldn't be a supported usecase. The fix should probably be also backported to previous releases.


Question: internalcfg, err := configutil.FetchInitConfigurationFromCluster(client, nil, logPrefix, false, false) if err == nil { fmt.Println() // add empty line to separate the FetchInitConfigurationFromCluster output from the command output <ins> // certificate renewal or expiration checking doesn't depend on a running cluster, which means the CertificatesDir // could be set to a value other than the default value or the value fetched from the cluster. // cfg.CertificatesDir could be empty if the default value is set to empty (not true today). if len(cfg.CertificatesDir) != 0 { klog.V(1).Infof("Overriding the cluster certificate directory with the value from command line flag --%s: %s", options.CertificatesDir, cfg.CertificatesDir) internalcfg.ClusterConfiguration.CertificatesDir = cfg.CertificatesDir } </ins> return internalcfg, nil } fmt.Printf("[%s] Error reading configuration from the Cluster. Falling back to default configurationnn", logPrefix)

When is set to a value other than the default value (/etc/kubernetes/pki/), and run on the working kubernetes controller node, the cert is not renewed as expected. This impacts the as well. or work as expected. Here is an example which show cert renew doesn't work, create a cert,and then renew the cert,check if the cert is renewed, It actually renewed the cert in the default path(/etc/kubernetes/pki/), The issue can be revealed by the test as well, NOTE This issue only exist with a working cluster since the code will fall back to use the cluster configuration when config file is not specified. And if the cluster cannot be connected with the , the will be honored, this is why the test can pass with the CI jobs. Check the log printed by this test pr: No response <details</details<detailslocal </details<details</details<details</details<details</details<details</details
/assign
/sig cluster-lifecycle
/area kubeadm
/triage accepted


Question: "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", <del> "--cluster-cidr=2001:101:115::/48", "--node-cidr-mask-size=64", </del> <ins> "--cluster-cidr=2001:db8::/64", "--node-cidr-mask-size=80", </ins> }, }, }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For IPv6, the node CIDR size is hard coded to /64. This causes an error bringing up the kube-controller, as the size must be for a smaller subnet (larger value) that what is used for the pod subnet, which is specified for some CNI plugins (e.g. Calico). In addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network (smaller value) than /66, so we cannot set it to /63, for example. What you expected to happen: The node CIDR size should be computed, based on the pod subnet specified, so that it can meet the requirements. How to reproduce it (as minimally and precisely as possible): Bring up a cluster using KubeAdm, specifying IPv6 and choose any pod subnet prefix desired. Anything else we need to know?: Note, there is a second requirement, that the difference between pod subnet size and node CIDR size be 16 or less (issue ). I'll be submitting a proposed PR to address this. Environment: Kubernetes version (use ): latest on master, using 1.9.2 in my case Cloud provider or hardware configuration: bare metal, single node OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux bxb-c2-77 4.4.0-109-generic -Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /area ipv6 /sig network
/assign pmichali


Question: asw.addVolumeToReportAsAttached(volumeName, nodeName) } <del> func (asw *actualStateOfWorld) RemoveNodeFromAttachUpdates(nodeName types.NodeName) error { asw.Lock() defer asw.Unlock() _, nodeToUpdateExists := asw.nodesToUpdateStatusFor[nodeName] if nodeToUpdateExists { delete(asw.nodesToUpdateStatusFor, nodeName) return nil } return fmt.Errorf("node %q does not exist in volumesToReportAsAttached list", nodeName) } </del> func (asw *actualStateOfWorld) AddVolumeNode( uniqueName v1.UniqueVolumeName, volumeSpec *volume.Spec, nodeName types.NodeName, devicePath string) (v1.UniqueVolumeName, error) { asw.Lock()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: Deployment running with read-only GCE PD. A node containing one of the deployment pods is deleted and recreated soon after. New pod is scheduled to the same node, but is stuck in ContainerCreating state. The node's kubelet logs show continuous "Volume is not yet attached according to node" errors. What you expected to happen: Pod recreated successfully. Anything else we need to know?: Cause by PR The issue happens as follows: When the node is deleted, GCE PD is detached from the cloud provider. AttachDetachController discovers the PD has been detached, and marks it as detached. The original pod still exists in the API server (i.e. before garbage collection), so AttachDetachController reattaches the PD, successfully. Before the node recovers, ADC tries to update the node status but finds that the node doesn't exist. Following the logic introduced in it never attempts to update the node status again. The node recovers, a new pod is scheduled to it, waits for a successful attach signal in the node status but never gets it. /cc /sig storage
/assign


Question: APIVersion: "v1", FieldPath:  "spec.containers[2]", }, <del> Reason:  "Started", Message: "some verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   3, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Started", Message:             "some verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               3, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"bar", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers[2]"}): type: 'Normal' reason: 'Started' some verbose message: 1`, expectUpdate: true,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: "//vendor/google.golang.org/api/googleapi:go_default_library", "//vendor/k8s.io/api/batch/v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", <ins> "//vendor/k8s.io/api/policy/v1beta1:go_default_library", </ins> "//vendor/k8s.io/api/rbac/v1beta1:go_default_library", "//vendor/k8s.io/api/storage/v1:go_default_library", "//vendor/k8s.io/api/storage/v1beta1:go_default_library",

When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


Question: fs.IPVar(&s.BindAddress, "insecure-bind-address", s.BindAddress, ""+ "The IP address on which to serve the --insecure-port (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).") fs.MarkDeprecated("insecure-bind-address", "This flag will be removed in a future version.") <ins> fs.Lookup("insecure-bind-address").Hidden = false </ins> fs.IntVar(&s.BindPort, "insecure-port", s.BindPort, ""+ "The port on which to serve unsecured, unauthenticated access. It is assumed "+

After merged, we should also show deprecated kube-apiserver flags.
/sig api-machinery


Question: flags.BoolVar(&TestContext.ListImages, "list-images", false, "If true, will show list of images used for runnning tests.") flags.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") <ins> flags.StringVar(&TestContext.ProgressReportURL, "progress-report-url", "", "The URL to POST progress updates to as the suite runs to assist in aiding integrations. If empty, no messages sent.") </ins> } // RegisterClusterFlags registers flags specific to the cluster e2e test suite.

What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } func TestEnsureLoadBalancerCreatesInternalLb(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: pvc.Spec.StorageClassName = &classWait pvc.Name = "claim02" }), <del> node: node2, </del> <ins> node: node3, </ins> }, "success-bad-and-good-node-affinity": { expectedMatch: "affinity-pv3",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Volume got bound to PV on node "kubernetes-minion-group-82c8" When the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the PV cache to point to the PVC: I think in , instead of continuing if the node affinity doesn't match, it should fail if the PV is prebound.
This causes two issues: On the second pass, the scheduler may choose a different node. Then in Assume/Bind volumes, we'll also prebind a second set of PVs. If the scheduler chose a different node on the second pass, and the PVCs became fully bound in between the predicates and assume volumes, then assume volumes will return success and the pod will get scheduled to the wrong node.


Question: Expect(pv.Spec.AccessModes).To(Equal(expectedAccessModes)) Expect(pv.Spec.ClaimRef.Name).To(Equal(claim.ObjectMeta.Name)) Expect(pv.Spec.ClaimRef.Namespace).To(Equal(claim.ObjectMeta.Namespace)) <ins> Expect(pv.Spec.MountOptions).To(Equal(class.MountOptions)) </ins> // Run the checker if t.pvCheck != nil {

for 1.8. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
Looking at the PR description, it seems that adds tests for ReclaimPolicy only. What about MountOptions?
those are coming also, I'll re-rename issue/point to another one
Since the feature is in 1.8, we need tests in 1.8. Adding this to 1.8 milestone.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
This issue is a 1.8 blocker. The PRs for to fix it, and , are approved/LGTM'd and awaiting merge. This issue can be closed as soon as those merge.


Question: } actualIDs := []string{} for _, healthDescription := range healthResponse.TargetHealthDescriptions { <del> if healthDescription.TargetHealth.Reason != nil { </del> <ins> if aws.StringValue(healthDescription.TargetHealth.State) == elbv2.TargetHealthStateEnumHealthy { actualIDs = append(actualIDs, *healthDescription.Target.Id) } else if healthDescription.TargetHealth.Reason != nil { </ins> switch aws.StringValue(healthDescription.TargetHealth.Reason) { case elbv2.TargetHealthReasonEnumTargetDeregistrationInProgress: // We don't need to count this instance in service if it is

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using a pod like the nginx-ing-cntlr with an AWS Network Load Balancer (NLB), and then draining the node its running on to force a migration to another node pool, does not reliably deregister the drained nodes from the NLB Target group. This causes the nodeport used by the NLB to remain open, and the drained node ends up continuing to funnel traffic for other pods cause of being used. This leads to connection timeouts for clients, and downtime on migrations. Switching to a Classic ELB, instead of an NLB, with connection draining enabled does work, and the instances that are drained are practically removed immediately from the instances targeted by the ELB, as expected. There seems to be some sort of issue with NLB's as this is not happening, or happening rarely (only saw it once in many tests but couldn't recreate it). The node + LB set seems to be working as I see in the controller-mgr logs that the nodes are updated as I went from 11 nodes to 8, after attempting to drain + migrate a nodepool of 3 nodes with nginx-ing-cntlr all running on them: However, this update does not seem to be translating into deregistering instances from the NLB. And i've waited minutes & hours in case timeouts were playing a hand, but that didn't seem to help. What you expected to happen: On node draining, with being used, that the NLB would deregister the instances as expected. How to reproduce it (as minimally and precisely as possible): Launch a Deployment + NLB service on a node and drain it to force it over to another node pool. Watch the instances in the AWS NLB Target group never get out of "healthy" status or be drained + removed. Anything else we need to know?: Environment: AWS EKS Kubernetes version (use ): 1.12 & 1.13 Cloud provider or hardware configuration: AWS OS (e.g: ): AWS Linux 2 Kernel (e.g.

): 4.14.123-111.109.amzn2.x8664 Install tools: Network plugin and version (if this is a network-related bug): core-dns v1.2.6 Others:
/sig aws
Related:
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: lastErr = fmt.Errorf("iscsi: failed to attach disk: Error: %s (%v)", string(out), err) continue } <del> exist = waitForPathToExist(devicePath, 10, iscsiTransport) </del> <ins> exist = waitForPathToExist(&devicePath, 10, iscsiTransport) </ins> if !exist { glog.Errorf("Could not attach disk: Timeout after 10s") // update last error

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-beta.1", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2017-06-08T13:08:32Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare-Metal, HP SE316M, 2x X5675, 144GB, iSCSI/RDMA/iSer OS: CentOS 7 Kernel: Linux at-host-18 4.9.31 SMP Wed Jun 7 22:40:40 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: I want to mount a iSCSI volume via iSER (RDMA over Infiniband) in this case the iSCSI iface is "iser" and not tcp, therefore the glob path for device mount is selected (), which then is passed to the fsck and mount commands without globbing the path (e.g. calling filepath.Glob) Therefore the iSCSI volume is not mounted and the pod cannot be started. Logs: What you expected to happen: To successfully mount the volume. How to reproduce it (as minimally and precisely as possible): Setup an iSCSI via RDMA (iSER) environment without multipath and try to mount a volume for a pod. Anything else we need to know: I believe it's just needed to add the globbing at line 259 like so: devicePaths = append(devicePaths, filepath.Glob(devicePath)) As I'm not a go programmer and have no experience with it, probably somebody could create a commit which I would gladly test!
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team.

You can find the team list and label list
/sig storage
I'll take a look.
/sig storage
on a local build that fixed the issue: (e.g. it compiles)
I don't want to add filepath.Glob to that general path because that line is also executed "tcp" case. I'm considering the fix below. Pretty similar.
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. )
Can we suppose that filepath.Glob(devicePath) always only return one device path if transport is "tcp"?
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. ) Thank you for above comment. I updated the logic. Could you confirm the fix on the PR?
Pretty sure that tcp transport will always only have one path - and it will exist (there's no glob in the check path)


Question: "k8s.io/kubernetes/pkg/api/v1" ) <ins> const ( MaxUint          = ^uint(0) MaxInt           = int(MaxUint >> 1) MaxTotalPriority = MaxInt MaxPriority      = 10 MaxWeight        = MaxInt / MaxPriority ) </ins> type Policy struct { metav1.TypeMeta // Holds the information to configure the fit predicate functions

Weight is an int (see ). Priority can take the range 1-10. Total priority for a node is sum of weight  priority for each priority function, and the sum is accumulated into an int (see ). So obviously the total priority can overflow if someone sets the weight(s) too high. The weights of all priority functions are compiled in (or specified in config file), so we can check to make sure maximum total priority can't overflow an int (take sum over all priority functions of 10  weight of that priority function, and make sure that's smaller than int(MaxUint 1) which according to StackOverflow is the way to find out what is max int (math package has max Int32, Int64, etc. but not max Int). Note that scheduler extender is OK because although the policy runs in a separate process, the weight is specified in scheduler config file (ExtenderConfig struct) and the priority is bounded 1-10 as usual.
We really should also make "score" be an abstract datatype rather than a raw int that is manipulated directly, so we can do things like make sure it is in the range 1-10 (or 0-10 if that's what we want); today the range is inconistent (we say 1-10, but sometimes use 0; probably we should change it to 0-9..).
I volunteer myself to do this.
I think we can check weight overflow in after we get all PriorityConfig. what do you think about this?
As for "score" I think we can use
Fixed by - closing.
Tim that PR was closed, not merged. I don't think this is fixed yet.
whoops, good catch, just removing milestone then. There was a shift that happened yesterday where a bunch of items suddenly came into the 1.6 query.
do we still want this fixed? I can re-pick this.
AFAIK it's still broken. cc/
Ok, I will open a PR tomorrow.


Question: )) # How to regenerate open-api code.  This emits a single file for all results. <del> $(OPENAPI_OUTFILE): $(OPENAPI_GEN) ./hack/run-in-gopath.sh $(OPENAPI_GEN)                                           --v $(KUBE_VERBOSE)                                                          --logtostderr                                                                -i $$(echo $(addprefix $(PRJ_SRC_PATH)/, $(OPENAPI_DIRS)) | sed 's/ /,/g')   -p $(PRJ_SRC_PATH)/$(OPENAPI_OUTPUT_PKG)                                     -O $(OPENAPI_BASENAME) </del> <ins> $(OPENAPI_OUTFILE): $(OPENAPI_GEN) $(OPENAPI_GEN) $(RUN_GEN_OPENAPI) </ins> # This calculates the dependencies for the generator tool, so we only rebuild # it when needed.  It is PHONY so that it always runs, but it only updates the

Right now on head () if you run it will end up having a modified file. This happened because some type has been changed and this file should have been generated before commit, and verification should have failed but they didn't.
This is not only a problem with openapi-gen. deepcopy-gen also has the same problem. I've a field to a type (autoscaling/v1) and verify-all passed for the commit (without adding generated code to it). those files will be generated on verify-codegen and it will not fail and the commit can be merged without generated code.
Yeah. So when the Makefile stuff originally went in those files were not checked in. Then we un-did that part, but I didn't re-add them to verify...


Question: } }) } <ins> } func TestShowHiddenMetric(t *testing.T) { registry := NewKubeRegistry(apimachineryversion.Info{ Major:      "1", Minor:      "15", GitVersion: "v1.15.0-alpha-1.12345", }) expectedMetricCount := 0 registry.MustRegister(alphaHiddenCounter) ms, err := registry.Gather() if len(ms) != expectedMetricCount { t.Errorf("Got %v metrics, Want: %v metrics", len(ms), expectedMetricCount) } showHidden.Store(true) defer showHidden.Store(false) registry.MustRegister(NewCounter( &CounterOpts{ Namespace:         "some_namespace", Name:              "test_alpha_show_hidden_counter", Subsystem:         "subsystem", StabilityLevel:    ALPHA, Help:              "counter help", DeprecatedVersion: &v114, }, )) expectedMetricCount = 1 ms, err = registry.Gather() if len(ms) != expectedMetricCount { t.Errorf("Got %v metrics, Want: %v metrics", len(ms), expectedMetricCount) } if err != nil { t.Fatalf("Gather failed %v", err) } </ins> }

The metrics framework requires the ability to optionally display hidden (deprecated) metrics. A boolean needs to be piped through the registry (possibly at initialization or possibly set later) which then needs to be passed to the Create function so that we can determine whether we need to actually create the metric. /sig instrumentation


Question: Expect(err).NotTo(HaveOccurred()) Expect(len(pods.Items)).Should(Equal(1)) }) <ins> It("will be set to nil if a patch removes the last pending initializer", func() { ns := f.Namespace.Name c := f.ClientSet podName := "to-be-patch-initialized-pod" framework.Logf("Creating pod %s", podName) // TODO: lower the timeout so that the server responds faster. _, err := c.CoreV1().Pods(ns).Create(newUninitializedPod(podName)) if err != nil && !errors.IsTimeout(err) { framework.Failf("expect err to be timeout error, got %v", err) } uninitializedPod, err := c.CoreV1().Pods(ns).Get(podName, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(uninitializedPod.Initializers).NotTo(BeNil()) Expect(len(uninitializedPod.Initializers.Pending)).Should(Equal(1)) patch := fmt.Sprintf(`{"metadata":{"initializers":{"pending":[{"$patch":"delete","name":"%s"}]}}}`, uninitializedPod.Initializers.Pending[0].Name) patchedPod, err := c.CoreV1().Pods(ns).Patch(uninitializedPod.Name, types.StrategicMergePatchType, []byte(patch)) Expect(err).NotTo(HaveOccurred()) Expect(patchedPod.Initializers).To(BeNil()) }) </ins> }) func newUninitializedPod(podName string) *v1.Pod {

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: if svcInfo.NodePort != 0 { // Hold the local port open so no other process can open it // (because the socket might open but it would never work). <del> lp := utilproxy.LocalPort{ Description: "nodePort for " + svcNameString, IP:          "", Port:        svcInfo.NodePort, Protocol:    protocol, </del> <ins> addresses, err := utilproxy.GetNodeAddresses(proxier.nodePortAddresses, proxier.networkInterfacer) if err != nil { glog.Errorf("Failed to get node ip address matching nodeport cidr: %v", err) continue </ins> } <del> if proxier.portsMap[lp] != nil { glog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] } else { socket, err := proxier.portMapper.OpenLocalPort(&lp) if err != nil { glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue </del> <ins> lps := make([]utilproxy.LocalPort, 0) for address := range addresses { lp := utilproxy.LocalPort{ Description: "nodePort for " + svcNameString, IP:          address, Port:        svcInfo.NodePort, Protocol:    protocol, } if utilproxy.IsZeroCIDR(address) { // Empty IP address means all lp.IP = "" lps = append(lps, lp) // If we encounter a zero CIDR, then there is no point in processing the rest of the addresses. break </ins> } <del> if lp.Protocol == "udp" { // TODO: We might have multiple services using the same port, and this will clear conntrack for all of them. // This is very low impact. The NodePort range is intentionally obscure, and unlikely to actually collide with real Services. // This only affects UDP connections, which are not common. // See issue: https://github.com/kubernetes/kubernetes/issues/49881 err := conntrack.ClearEntriesForPort(proxier.exec, lp.Port, isIPv6, v1.ProtocolUDP) </del> <ins> lps = append(lps, lp) } // For ports on node IPs, open the actual port and hold it. for _, lp := range lps { if proxier.portsMap[lp] != nil { glog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] } else { socket, err := proxier.portMapper.OpenLocalPort(&lp) </ins> if err != nil { <del> glog.Errorf("Failed to clear udp conntrack for port %d, error: %v", lp.Port, err) </del> <ins> glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue } if lp.Protocol == "udp" { // TODO: We might have multiple services using the same port, and this will clear conntrack for all of them. // This is very low impact. The NodePort range is intentionally obscure, and unlikely to actually collide with real Services. // This only affects UDP connections, which are not common. // See issue: https://github.com/kubernetes/kubernetes/issues/49881 err := conntrack.ClearEntriesForPort(proxier.exec, lp.Port, isIPv6, v1.ProtocolUDP) if err != nil { glog.Errorf("Failed to clear udp conntrack for port %d, error: %v", lp.Port, err) } </ins> } <ins> replacementPortsMap[lp] = socket </ins> } <del> replacementPortsMap[lp] = socket </del> } if hasEndpoints {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-proxy was still binding to ::{port} for a node port service after it was started with the --nodeport-addresses argument What you expected to happen: kube-proxy to only bind to interfaces in the CIDR ranges specified by --nodeport-addresses. How to reproduce it (as minimally and precisely as possible): Start kube-proxy with --nodeport-addresses "127.0.0.1,::1" Create a service of type NodePort Observe kube-proxy bind to all listening interfaces Anything else we need to know?: In the proposal for the --nodeport-addresses feature, it specifies being able to have other services answering or listening on a given port. Since kube-proxy binds to all interfaces, this is still not possible so it appears one of the initial intents of the proposal was net met with the existing solution. [] Doesn't set an interface. Given that nodeport-addresses is specified as CIDR ranges extra work must be done to determine which IP's on a host exist that could be bound to. Alternatively, a flag such as --nodeport-addresses-bind could allow an operator to set an explicit value. To test functionality I was able to hardcode an address here and observe the desired behavior could be achieved. Environment: Kubernetes version:Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom fabricated Others: As expected, this behaves the same way regardless of iptables or ipvs being used as the forwarding mechanism.
/sig network
/assign
Nice catch! I will fix it ASAP~
Please check


Question: } } } <ins> func BenchmarkPodMatchesScopeFunc(b *testing.B) { pod, _ := toExternalPodOrError(makePod("p1", "high-priority", api.ResourceList{api.ResourceCPU: resource.MustParse("1")}, api.PodRunning)) tests := []struct { name     string selector corev1.ScopedResourceSelectorRequirement }{ { name: "PriorityClass selector w/o operator", selector: corev1.ScopedResourceSelectorRequirement{ ScopeName: corev1.ResourceQuotaScopePriorityClass, }, }, { name: "PriorityClass selector w/ 'Exists' operator", selector: corev1.ScopedResourceSelectorRequirement{ ScopeName: corev1.ResourceQuotaScopePriorityClass, Operator:  corev1.ScopeSelectorOpExists, }, }, { name: "BestEfforts selector w/o operator", selector: corev1.ScopedResourceSelectorRequirement{ ScopeName: corev1.ResourceQuotaScopeBestEffort, }, }, { name: "BestEfforts selector w/ 'Exists' operator", selector: corev1.ScopedResourceSelectorRequirement{ ScopeName: corev1.ResourceQuotaScopeBestEffort, Operator:  corev1.ScopeSelectorOpExists, }, }, } for _, tt := range tests { b.Run(tt.name, func(b *testing.B) { for n := 0; n < b.N; n++ { _, _ = podMatchesScopeFunc(tt.selector, pod) } }) } } func mockListerForResourceFunc(listerForResource map[schema.GroupVersionResource]cache.GenericLister) quota.ListerForResourceFunc { return func(gvr schema.GroupVersionResource) (cache.GenericLister, error) { lister, found := listerForResource[gvr] if !found { return nil, fmt.Errorf("no lister found for resource %v", gvr) } return lister, nil } } func newGenericLister(groupResource schema.GroupResource, items []runtime.Object) cache.GenericLister { store := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{"namespace": cache.MetaNamespaceIndexFunc}) for _, item := range items { store.Add(item) } return cache.NewGenericLister(store, groupResource) } func makePod(name, pcName string, resList api.ResourceList, phase api.PodPhase) *api.Pod { return &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: api.PodSpec{ PriorityClassName: pcName, Containers: []api.Container{ { Resources: api.ResourceRequirements{ Requests: resList, Limits:   resList, }, }, }, }, Status: api.PodStatus{ Phase: phase, }, } } </ins>

If a Pod is subjected to a with "PriortyClass" as its scope, the quota's calculation is incorrect. See repo steps for more details. with "PriortyClass" as its scope can be calculated at all times. Create a PriorityClass named "p1"Create a quota with as its scope. Note: w/o specifying , this issue won't happenCreate a Job and wait for its completion.Check quota's usage calculation Here is incorrect - it should be . Worse calculation may happen if I create a regular pod and then delete the Job:The eventual result is totally incorrect. No response master, and possibly all versions. kind <details</details<details</details<details</details<details</details
/sig api-machinery /kind bug
/triage accepted
I would like to take this up!
is already in flight for this


Question: pieces := strings.Split(part, ":") if len(pieces) < 1 || len(pieces) > 2 { glog.Infof("Bad port spec: %s", part) <del> continue </del> <ins> return nil, fmt.Errorf("Bad port spec: %s", part) </ins> } host := 0 container := 0

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: cd $(TEMP_DIR) && echo "ENV ETCD_UNSUPPORTED_ARCH=$(ARCH)" >> Dockerfile endif <del> # Replace BASEIMAGE with the real base image cd $(TEMP_DIR) && sed -i.bak 's|BASEIMAGE|$(BASEIMAGE)|g' Dockerfile cd $(TEMP_DIR) && sed -i.bak 's|RUNNERIMAGE|$(RUNNERIMAGE)|g' Dockerfile </del> # And build the image <del> docker build --pull -t $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(TEMP_DIR) </del> <ins> docker build  --pull  -t $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG)  --build-arg BASEIMAGE=$(BASEIMAGE)  --build-arg RUNNERIMAGE=$(RUNNERIMAGE)  $(TEMP_DIR) </ins> push: build docker tag $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(MANIFEST_IMAGE)-$(ARCH):$(IMAGE_TAG)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Building the etcd integration-test-image fails with the following err: Failure Log: Expected the image build to pass. k/k cluster/images/etcd build-integration-test-image Change that probably broke it: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig etcd
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig etcd Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
/sig api-machinery
/cc /cc
/cc
/assign /triage accepted
is working on this in .


Question: type EnvVar struct { // Required: This must be a C_IDENTIFIER. Name string `json:"name" description:"name of the environment variable; must be a C_IDENTIFIER"` <del> // Optional: defaults to "". </del> <ins> // Optional: No more than one of the following may be set. // Optional: Defaults to "" </ins> Value string `json:"value,omitempty" description:"value of the environment variable; defaults to empty string"` <del> // Optional: specify a source the value of this var should come from. </del> <ins> // Optional: Specifies a source the value of this var should come from. </ins> ValueFrom *EnvVarSource `json:"valueFrom,omitempty" description:"source for the environment variable's value; cannot be used if value is not empty"` } // EnvVarSource represents a source for the value of an EnvVar. <del> // Only one of its members may be specified. </del> type EnvVarSource struct { <del> // Selects a field of the pod; only name and namespace are supported. FieldPath *ObjectFieldSelector `json:"fieldPath,omitempty" description:"selects a field of the pod; only name and namespace are supported"` </del> <ins> // Required: Selects a field of the pod; only name and namespace are supported. FieldPath *ObjectFieldSelector `json:"fieldPath" description:"selects a field of the pod; only name and namespace are supported"` </ins> } // ObjectFieldSelector selects an APIVersioned field of an object. type ObjectFieldSelector struct { <del> // The API version the FieldPath is written in terms of. APIVersion string `json:"apiVersion,omitempty" description="The API version that FieldPath is written in terms of"` // The path of the field to select in the specified API version FieldPath string `json:"fieldPath,omitempty" description="The path of the field to select in the specified API version"` </del> <ins> // Optional: Version of the schema the FieldPath is written in terms of, defaults to "v1" APIVersion string `json:"apiVersion,omitempty" description:"version of the schema that fieldPath is written in terms of; defaults to v1"` // Required: Path of the field to select in the specified API version FieldPath string `json:"fieldPath" description:"path of the field to select in the specified API version"` </ins> } // HTTPGetAction describes an action based on HTTP Get requests.

PR had a few errors in its API objects, where the "description" field errors in its JSON declarations were missing. But, the "verify-" presubmit check didn't flag them. This should be investigated and fixed.
Could you please take a look at this?


Question: for i := 0; i < 3; i++ { framework.Logf("PD Read/Writer Iteration #%v", i) By("submitting host0Pod to kubernetes") <ins> host0Pod = testPDPod([]string{diskName}, host0Name, false /* readOnly */, numContainers) </ins> _, err = podClient.Create(host0Pod) framework.ExpectNoError(err, fmt.Sprintf("Failed to create host0Pod: %v", err))

PR moved some PD tests to flaky to resolve issue (and duplicate ). These tests started failing due to PR . is working on a patch to de-flake the tests. This issue is to track the fix and should be closed when: PD tests have been deflaked (verify by running the "" e2e tests back to back for a couple of hours to verify the issue is fixed). flaky label has been removed from the "" tests that were marked flaky in CC
how to watch the submit queue and find if the flake is gone?
I don't know. is there any way can verify whether test is no longer flaky?
I'm not sure what the publicly accessible link for test history is. I took a look for you: Pod Disks should schedule a pod w/ a RW PD shared between multiple containers, write to PD, delete pod, verify contents, and repeat in rapid succession Failed 3 times in the last 30 runs. Stability: 90 % Pod Disks should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession Failed 1 times in the last 30 runs. Stability: 96 % I have a PR pending that enables the new attach/detach controller and I was able to run the tests back to back overnight without any failures. So once that gets merged, we can take a look at moving these tests back.
Something in PD attach/detach affects also 'Dynamic provisioning' e2e test, a PD is still attached to a node ~15 minutes after appropriate pod was deleted and thus deleter can't delete it. From controller log:
please take a look at https://k8s-
thanks so at least one of the flake has a different error than "container not found" In one, the error is
^^ the above seems like a possible bug in exec
^
The error to me indicates that the original container is no longer around as exec can't find the ipc namespace fd. Are we sure the container that we are trying to join is still around?
my memory is rusty but did you have an idea to randomize the container id in this pd test?
see the info here:
Close we will track this with


Question: type Client struct { service   *bigquery.Service <del> token     *oauth.Token </del> <ins> token     *oauth2.Token </ins> datasetId string tableId   string } // Helper method to create an authenticated connection. <del> func connect() (*oauth.Token, *bigquery.Service, error) { </del> <ins> func connect() (*oauth2.Token, *bigquery.Service, error) { </ins> if *clientId == "" { return nil, nil, fmt.Errorf("no client id specified") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted


Question: nodestatus.PIDPressureCondition(kl.clock.Now, kl.evictionManager.IsUnderPIDPressure, kl.recordNodeStatusEvent), nodestatus.ReadyCondition(kl.clock.Now, kl.runtimeState.runtimeErrors, kl.runtimeState.networkErrors, validateHostFunc, kl.containerManager.Status, kl.recordNodeStatusEvent), nodestatus.VolumesInUse(kl.volumeManager.ReconcilerStatesHasBeenSynced, kl.volumeManager.GetVolumesInUse), <ins> nodestatus.RemoveOutOfDiskCondition(), </ins> // TODO(mtaufen): I decided not to move this setter for now, since all it does is send an event // and record state back to the Kubelet runtime object. In the future, I'd like to isolate // these side-effects by decoupling the decisions to send events and partial status recording

What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3


Question: - --memory={{ base_eventer_memory }} - --extra-memory={{ eventer_memory_per_node }}Ki - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=eventer - --poll-period=300000 - --estimator=exponential

I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue?

This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.


Question: // - generic pruning of unknown fields following a structural schema. type unstructuredSchemaCoercer struct { dropInvalidMetadata bool <ins> repairGeneration    bool </ins> structuralSchemas     map[string]*structuralschema.Structural structuralSchemaGK    schema.GroupKind

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: "//vendor/github.com/spf13/pflag:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/util/flag:go_default_library", </ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", "//vendor/k8s.io/client-go/tools/bootstrap/token/api:go_default_library", "//vendor/k8s.io/client-go/tools/bootstrap/token/util:go_default_library",

/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Question: return pod.Status.Phase == v1.PodFailed } <del> // isTerminated returns true if pod's deletion Timestamp has been set func isTerminated(pod *v1.Pod) bool { </del> <ins> // isTerminating returns true if pod's DeletionTimestamp has been set func isTerminating(pod *v1.Pod) bool { </ins> return pod.DeletionTimestamp != nil } // isHealthy returns true if pod is running and ready and has not been terminated func isHealthy(pod *v1.Pod) bool { <del> return isRunningAndReady(pod) && !isTerminated(pod) </del> <ins> return isRunningAndReady(pod) && !isTerminating(pod) </ins> } // newControllerRef returns an ControllerRef pointing to a given StatefulSet.

/assign flakes in PR jobs more than once per day The can filter down to searching for results for a single test:
This test continues to flake PRs at least once per day:
https://k8s-
looking at it


Question: desiredReplicas = scaleUpLimit case hpa.Spec.MinReplicas != nil && desiredReplicas < *hpa.Spec.MinReplicas: // make sure we aren't below our minimum <del> setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionTrue, "TooFewReplicas", "the desired replica count was less than the minimum replica count") </del> <ins> var statusMsg string if desiredReplicas == 0 { statusMsg = "the desired replica count was zero" } else { statusMsg = "the desired replica count was less than the minimum replica count" } setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionTrue, "TooFewReplicas", statusMsg) </ins> desiredReplicas = *hpa.Spec.MinReplicas case desiredReplicas == 0: //  never scale down to 0, reserved for disabling autoscaling

What happened: Wrong HPA condition - ScalingLimited have one rc with 1 replica(current have 1 pod) have one hpa with spec.minReplicas=1 and spec.maxReplicas=10 HPA condition ScalingLimited is true and described by below, I expect ScalingLimited=false at that time. What you expected to happen: I expect ScalingLimited=false at above time How to reproduce it (as minimally and precisely as possible): kubernetes with DNS api server with --runtime-config="api/all=true" binaries and images, deploy heapster, influxdb and grafana in kubernetes a rc a v2 hpa hpa conditions by I expect ScalingLimited of hpa conditions should be: Addition info:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
could you please take a look.
/sig autoscaling
It looks, at first glance, like what's happening is that since CPU usage is zero, we're calculating a desired replica count of 0, which is less that the minimum (1), so we're getting . We might want to detect this condition and provide another status message that's clearer. does that make sense on your end? Does the problem persist if CPU usage is non-zero?
Yes, the problem only persist if CPU usage is zero, and not exist when CPU usage is non-zero. I will keep this issue open if you want to improve and let it more clearer while CPU usage is zero. Thank you.
If this is still an open issue, I'd love to take a shot at submitting a pr for it. To confirm, the desired behavior is that if CPU usage is zero, and the desired replica count is 0, which is less than the minimum (1), should still be , but the status message should make it clear that it is because the CPU usage is zero?
I'd be happy to review such a PR :-). Yep! Having zero usage is a bit of a special case, so it's probably worth letting people know explicitly.
Wonderful! Working on it :)


Question: claimSize:    "5Gi", expectedSize: "5Gi", nodeName:     node.Name, <ins> attach:       true, </ins> } }

brought this up during the CSI Standup. It seems to be true, in the test config we do not set which is required to bring up the pod for consuming the volume. The code where is checked: The test config used where is not set to true: /kind bug /sig storage /assign /cc


Question: type PersistentVolumeClaimList struct { TypeMeta `json:",inline"` <del> ListMeta `json:"metadata,omitempty"` </del> <ins> ListMeta `json:"metadata,omitempty" description:"standard list metadata; see http://docs.k8s.io/api-conventions.md#types-kinds` </ins> Items    []PersistentVolumeClaim `json:"items,omitempty" description: "a list of persistent volume claims"` }

PR had a few errors in its API objects, where the "description" field errors in its JSON declarations were missing. But, the "verify-" presubmit check didn't flag them. This should be investigated and fixed.
Could you please take a look at this?