Question: return pod } <del> // configureHugePages attempts to allocate 100Mi of 2Mi hugepages for testing purposes </del> <ins> // configureHugePages attempts to allocate 10Mi of 2Mi hugepages for testing purposes </ins> func configureHugePages() error { <del> err := exec.Command("/bin/sh", "-c", "echo 50 > /proc/sys/vm/nr_hugepages").Run() </del> <ins> err := exec.Command("/bin/sh", "-c", "echo 5 > /proc/sys/vm/nr_hugepages").Run() </ins> if err != nil { return err }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: -tests Since when has it been failing: Unsure Testgrid link: Reason for failure: Unable to pre allocate 50x2MiB huge pages, probably due to memory pressure caused by kernel level caching. If requesting an substantial amount of huge page memory (like the tests), it is recommended to to it as early after boot as possible, or by using the kernel cmd line. Eg. this one: Anything else we need to know: will fix this issue, but in awaiting for that, we can just decrease the page count in the tests.
/sig node /cc derekwaynecarr
I think the reason of railure should be because a single hugepage of x86 is 2MiB as and a single page is 4KiB, not 2KiB.
Yes, is correct. Thanks for catching the typo :smile:
Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. This is probably because the kernel is caching a lot of stuff, making it hard for the kernel to fix it. Ill do some testing with dropping the caches (no user facing stuff will happen, only kernel level) before trying to pre allocate huge pages. /reopen
Reopened this issue. <detailsIn response to : Looks like we still have some flakes in the huge page test, even tho the test now only requests of huge page memory. Happens way less frequently than previously tho. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" "net" <ins> "sync" "sync/atomic" </ins> "testing" "time" )

What happened: my kubelet restart unexpectedly,i find log info has more "Failed to list" info and has fatal error: go of nil func value,and then kubelet restart Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"elasticsearch-token-99wc2": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] failed 5 attempts to update node lease, will retry after 10s Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"default-token-6wh7n": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-rules": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"logging"/"service-registry": Failed to list v1.Secret: Get https://vip- Feb 02 17:46:25 mgt15 kubelet[6914]: E0202 17:46:25. 6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:30 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] Failed to list v1beta1.RuntimeClass: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30. 6914 ] object-"logging"/"telegraf-elasticsearch": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:31 mgt15 kubelet[6914]: E0202 17:46:30.

6914 ] object-"monitoring"/"prometheus-center": Failed to list v1.ConfigMap: Get https://vip- Feb 02 17:46:32 mgt15 kubelet[6914]: fatal error: go of nil func value Feb 02 17:46:32 mgt15 kubelet[6914]: goroutine 357 [running]: Feb 02 17:46:32 mgt15 kubelet[6914]: (closableConn).Close(0x4002193f00, 0x0, 0x0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Dialer).CloseAll(0x4000bbaea0) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).updateNodeStatus(0x40010ce000, 0x54d0978, 0x40010ce648) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- +0xf0 Feb 02 17:46:32 mgt15 kubelet[6914]: (Kubelet).syncNodeStatus(0x40010ce000) Feb 02 17:46:32 mgt15 kubelet[6914]: /workspace/anago-v1.14.3- What you expected to happen: kubelet do not restart How to reproduce it (as minimally and precisely as possible): when have more info of "Failed to list" Environment: Kubernetes version (use ):v1.14.3 OS (e.g: ):Ubuntu 18.04.3 LTS Kernel (e.g.

):4.15.0-55-generic Network plugin and version (if this is a network-related bug):calico3.8.2
/sig node /wg node /committee node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node /wg node /committee node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix is in
/sig auth
/assign
thankswhat version is this bug been fixed?
The fix is not yet merged, but I plan to backport the fix to 1.15.x, 1.16.x, 1.17.x (the exact patch versions depend on when the backports are merged)


Question: detect-jessie-image ;; *) <del> echo "Please specify AWS_IMAGE directly (distro ${KUBE_OS_DISTRIBUTION} not recognized)" </del> <ins> echo "Please specify AWS_IMAGE directly (distro ${OS_DISTRIBUTION} not recognized)" </ins> exit 2 ;; esac

If e.g. coreos doesn't work with kube-up, we should point users to an installation method that works. Also consider offering hints like "vivid no longer gets security updates, consider using wily or official"


Question: - mountPath: /etc/kubernetes/ name: addons readOnly: true <ins> - mountPath: /var/log name: varlog readOnly: false </ins> volumes: - hostPath: path: /etc/kubernetes/ name: addons <ins> - hostPath: path: /var/log name: varlog </ins>

Addon Manager was broken a few times recently by some indirect-related changes(like ) made in the code base. Addon Manager's logs would be very helpful for narrowing down the problem. And I don't think there is an obvious way to retrieve Addon Manager's logs for Jenkins tests currently. This can be achieved in the same way as what we did for the components like and (mounting /var/log). I will try it through this way for addon manager. cc


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "testing" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/policy" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/fake" ) func TestEviction(t *testing.T) { testcases := []struct { name     string pdbs     []runtime.Object pod      *api.Pod eviction *policy.Eviction expectError   bool expectDeleted bool }{ { name:          "no pdbs, unscheduled pod, nil delete options, deletes immediately", pdbs:          nil, pod:           validNewPod(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: true, }, { name:          "no pdbs, scheduled pod, nil delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, empty delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: &metav1.DeleteOptions{}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, graceless delete options, deletes immediately", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "matching pdbs with no disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:    &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectError: true, }, { name: "matching pdbs with disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 1}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "non-matching pdbs", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"b": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testContext := genericapirequest.WithNamespace(genericapirequest.NewContext(), metav1.NamespaceDefault) storage, _, _, server := newStorage(t) defer server.Terminate(t) defer storage.Store.DestroyFunc() if tc.pod != nil { if _, err := storage.Create(testContext, tc.pod, nil, &metav1.CreateOptions{}); err != nil { t.Error(err) } } client := fake.NewSimpleClientset(tc.pdbs...) evictionRest := newEvictionStorage(storage.Store, client.Policy()) _, err := evictionRest.Create(testContext, tc.eviction, nil, &metav1.CreateOptions{}) if (err != nil) != tc.expectError { t.Errorf("expected error=%v, got %v", tc.expectError, err) return } if tc.expectError { return } if tc.pod != nil { existingPod, err := storage.Get(testContext, tc.pod.Name, &metav1.GetOptions{}) if tc.expectDeleted { if !apierrors.IsNotFound(err) { t.Errorf("expected to be deleted, lookup returned %#v", existingPod) } return } else if apierrors.IsNotFound(err) { t.Errorf("expected graceful deletion, got %v", err) return } if err != nil { t.Errorf("%#v", err) return } if existingPod.(*api.Pod).DeletionTimestamp == nil { t.Errorf("expected gracefully deleted pod with deletionTimestamp set, got %#v", existingPod) } } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set.

Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


Question: <del> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package state import ( "fmt" "sync" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager/errors" ) var _ CheckpointState = &stateCheckpoint{} // CheckpointState interface provides to get and store state type CheckpointState interface { GetOrCreate() (ClaimInfoStateList, error) Store(ClaimInfoStateList) error } // ClaimInfoState is used to store claim info state in a checkpoint // +k8s:deepcopy-gen=true type ClaimInfoState struct { // ClaimUID is an UID of the resource claim ClaimUID types.UID // ClaimName is a name of the resource claim ClaimName string // Namespace is a claim namespace Namespace string // PodUIDs is a set of pod UIDs that reference a resource PodUIDs sets.Set[string] // DriverState contains information about all drivers which have allocation // results in the claim, even if they don't provide devices for their results. DriverState map[string]DriverState } // DriverState is used to store per-device claim info state in a checkpoint // +k8s:deepcopy-gen=true type DriverState struct { Devices []Device } // Device is how a DRA driver described an allocated device in a claim // to kubelet. RequestName and CDI device IDs are optional. // +k8s:deepcopy-gen=true type Device struct { PoolName     string DeviceName   string RequestNames []string CDIDeviceIDs []string } type stateCheckpoint struct { sync.RWMutex checkpointManager checkpointmanager.CheckpointManager checkpointName    string } // NewCheckpointState creates new State for keeping track of claim info  with checkpoint backend func NewCheckpointState(stateDir, checkpointName string) (*stateCheckpoint, error) { if len(checkpointName) == 0 { return nil, fmt.Errorf("received empty string instead of checkpointName") } checkpointManager, err := checkpointmanager.NewCheckpointManager(stateDir) if err != nil { return nil, fmt.Errorf("failed to initialize checkpoint manager: %v", err) } stateCheckpoint := &stateCheckpoint{ checkpointManager: checkpointManager, checkpointName:    checkpointName, } return stateCheckpoint, nil } // get state from a checkpoint and creates it if it doesn't exist func (sc *stateCheckpoint) GetOrCreate() (ClaimInfoStateList, error) { sc.Lock() defer sc.Unlock() checkpoint := NewDRAManagerCheckpoint() err := sc.checkpointManager.GetCheckpoint(sc.checkpointName, checkpoint) if err == errors.ErrCheckpointNotFound { sc.store(ClaimInfoStateList{}) return ClaimInfoStateList{}, nil } if err != nil { return nil, fmt.Errorf("failed to get checkpoint %v: %w", sc.checkpointName, err) } return checkpoint.Entries, nil } // saves state to a checkpoint func (sc *stateCheckpoint) Store(claimInfoStateList ClaimInfoStateList) error { sc.Lock() defer sc.Unlock() return sc.store(claimInfoStateList) } // saves state to a checkpoint, caller is responsible for locking func (sc *stateCheckpoint) store(claimInfoStateList ClaimInfoStateList) error { checkpoint := NewDRAManagerCheckpoint() checkpoint.Entries = claimInfoStateList err := sc.checkpointManager.CreateCheckpoint(sc.checkpointName, checkpoint) if err != nil { return fmt.Errorf("could not save checkpoint %s: %v", sc.checkpointName, err) } return nil } </del>

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: if len(args) > 0 && len(o.Selector) > 0 { return cmdutil.UsageErrorf(cmd, "error: cannot specify both a node name and a --selector option") } <del> if len(args) > 0 && len(args) != 1 { return cmdutil.UsageErrorf(cmd, fmt.Sprintf("USAGE: %s [flags]", cmd.Use)) } </del> o.DryRun = cmdutil.GetDryRunFlag(cmd)

They should be consistent with all other commands that allow multiple arguments (since they already allow label selectors). should cordon all of them, not return an error.
Hi , can I take this if no one has been working on it?
Please also refer . These two and should take multiple arguments as it is very common to get subset of nodes in this way . So it would be very nice if these would work:


Question: lines := strings.Split(raw, linebreak) for _, line := range lines { trimmed := strings.Trim(line, " nt") <ins> if len(trimmed) > 0 && trimmed[0] != '_' { out.WriteString(" ") } </ins> out.WriteString(trimmed) <del> out.WriteString(" ") </del> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kubectl help description can't display underscore properly. Example 1: The environment variable displayed but missing underscore. Example 2: The environment variable with an additional space before underscore. What you expected to happen: Display underscore properly without missing or additional space. How to reproduce it (as minimally and precisely as possible): Use kubectl and run following example: Anything else we need to know?: The description are correctly in code base, but display with wrong message format. Environment: Kubernetes version (use ): and also linux version kubectl <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
I found the same problem PROPERTY _NAME the same as above you paste in
, I would like to take this up :smile:
oops, it's already there:


Question: // resourceVersion may be used to specify what version to begin watching, // which should be the current resourceVersion, and no longer rv+1 // (e.g. reconnecting without missing any updates). <ins> // If resource version is "0", this interface will get current object at given key // and send it in an "ADDED" event, before watch starts. </ins> Watch(ctx context.Context, key string, resourceVersion string, p SelectionPredicate) (watch.Interface, error) // WatchList begins watching the specified key's items. Items are decoded into API

Currently, Watch request with resourceVersion=0 results in: returning a single "watch event" for every object existing "now" starts watching from that point This is pretty misleading (e.g. it's different that what we have in etcd, where there is no first step from above). In the v2 api, we should probably eliminate the first stop (or even disallow watch requests with resourceVersion = 0).
/cc
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/remove-lifecycle rotten /lifecycle frozen
What do you mean by v2 API? Are we ever going to have a v2 of the machiery and semantics?
In 2015 I thought it's not ridiculous. I'm not sure I believe in it now. Moreover, with we will actually be taking benefits of semantic like this, so we should close this.


Question: }, }, } <del> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, vmName, diskName) </del> <ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s, %s)", nodeResourceGroup, vmName, diskName, diskURI) </ins> ctx, cancel := getContextWithCancel() defer cancel()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add more logging in azure disk attach/detach What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
The issue message here doesn't say why a change was desired. The PR "Original logging is not clear when dealing with managed disk, since diskName is empty". That's more informative.


Question: return h } <del> /* utf16BytesToString converts UTF-16 encoded bytes, in big or little endian byte order, to a UTF-8 encoded string. http://stackoverflow.com/a/15794113 */ func utf16BytesToString(b []byte, o binary.ByteOrder) string { utf := make([]uint16, (len(b)+(2-1))/2) for i := 0; i+(2-1) < len(b); i += 2 { utf[i/2] = o.Uint16(b[i:]) } if len(b)/2 < len(utf) { utf[len(utf)-1] = utf8.RuneError } return string(utf16.Decode(utf)) } </del>

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: }, }, { <ins> // matching pods spread as 2/1/0/3. // With maxSkew=2, counts change internally to 2/1/1/3. name: "one constraint on node, all 4 nodes are candidates, maxSkew=2", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(2, v1.LabelHostname, v1.ScheduleAnyway, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-d1").Node("node-d").Label("foo", "").Obj(), st.MakePod().Name("p-d2").Node("node-d").Label("foo", "").Obj(), st.MakePod().Name("p-d3").Node("node-d").Label("foo", "").Obj(), }, nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label(v1.LabelHostname, "node-a").Obj(), st.MakeNode().Name("node-b").Label(v1.LabelHostname, "node-b").Obj(), st.MakeNode().Name("node-c").Label(v1.LabelHostname, "node-c").Obj(), st.MakeNode().Name("node-d").Label(v1.LabelHostname, "node-d").Obj(), }, failedNodes: []*v1.Node{}, want: []framework.NodeScore{ {Name: "node-a", Score: 60},  // +20, compared to maxSkew=1 {Name: "node-b", Score: 100}, // +20, compared to maxSkew=1 {Name: "node-c", Score: 100}, {Name: "node-d", Score: 20}, // +20, compared to maxSkew=1 }, }, { // matching pods spread as 4/3/2/1. // With maxSkew=3, counts change internally to 4/3/2/2. name: "one constraint on node, all 4 nodes are candidates, maxSkew=3", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(3, v1.LabelHostname, v1.ScheduleAnyway, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a3").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a4").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-b2").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-b3").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-c1").Node("node-c").Label("foo", "").Obj(), st.MakePod().Name("p-c2").Node("node-c").Label("foo", "").Obj(), st.MakePod().Name("p-d1").Node("node-d").Label("foo", "").Obj(), }, nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label(v1.LabelHostname, "node-a").Obj(), st.MakeNode().Name("node-b").Label(v1.LabelHostname, "node-b").Obj(), st.MakeNode().Name("node-c").Label(v1.LabelHostname, "node-c").Obj(), st.MakeNode().Name("node-d").Label(v1.LabelHostname, "node-d").Obj(), }, failedNodes: []*v1.Node{}, want: []framework.NodeScore{ {Name: "node-a", Score: 42},  // +28 compared to maxSkew=1 {Name: "node-b", Score: 71},  // +29 compared to maxSkew=1 {Name: "node-c", Score: 100}, // +29 compared to maxSkew=1 {Name: "node-d", Score: 100}, }, }, { </ins> // matching pods spread as 4/2/1/~3~ (node4 is not a candidate) name: "one constraint on node, 3 out of 4 nodes are candidates", pod: st.MakePod().Name("p").Label("foo", "").

Spin-off from What would you like to be : Use maxSkew in TopologySpreadingContraint to affect relative weight of the constraint. Exact formula to be discussed. Some ideas: , intuitive but too coarse-grained at low numbers. Goes: 100%, 50%, 33%, 25%, 20% some linearity. For , goes: 100%, 66%, 50%, 40%, 33% . For , goes: 100%, 75%, 60%, 50%, 42% . Goes: 100%, 70%, 57%, 50%, 44% Why is this needed: When using for , it will be useful for operators to control how strong the constraints should be compared to explicit user requests. Pods would be able to control the weight of constraints per-topology.
/sig scheduling
1 maybe the most compatible option with the current zone:node ratio (2:1):
Not quite. The ratio is correct, but the weight is different. original: prop 1:
I realized that using as weight is quite counterintuitive. Instead, I propose that we use to water down the counts per topology. This way, the behavior still doesn't change for . But as increases, counts become more similar. For example, given the counts and , we get the new counts: . Then we proceed with the counts normalization as usual. Note that when the minimum count is zero in a topology, this procedure has the effect of giving the max score (100) for nodes (or zones) that would satisfy the condition. That won't be true when the minimum count is not zero, but enforcing it would be quite costly. This proposal still gets us closer to that and reduce score differentiation among nodes within the skew. When using constraints against multiple topologies, another side effect is that topologies with bigger would have a reduced effect in the overall score.
That sounds better than the initial proposal which seemed to deviate from the original semantics of the API.
WDYT ?
Sounds good to me. Per constraint on large-scope topology (e.g., zone), I assume it will still be credited with the matching number on its sub topologies (e.g., node), right? (It's just we're using the newCount formula - by taking the maxSkew specified on its constraint into consideration.)

Note that this doesn't replace increasing the weight as described in the parent issue. I think the issue title needs updating.
Understood. Default weight is tracked in another issue. This one talks about Scoring.
Right, just wanted to explicitly mention that to clarify the overall plan since this issue started in the context of increasing the default weight :)
Could you rename that one and clarify that it depends on this issue being resolved first?
can you clarify why that issue is dependent on this one? aren't we going to increase the weight regardless?
I was referring to . But you are right, it doesn't depend on this. kubernetes/enhancements does


Question: return nil, err } informerFactory := handle.SharedInformerFactory() <del> podLister := handle.SnapshotSharedLister().Pods() </del> serviceLister := informerFactory.Core().V1().Services().Lister() <del> priorityMapFunction, priorityReduceFunction := priorities.NewServiceAntiAffinityPriority(podLister, serviceLister, args.AntiAffinityLabelsPreference) </del> return &ServiceAffinity{ <del> sharedLister:           handle.SnapshotSharedLister(), serviceLister:          serviceLister, priorityMapFunction:    priorityMapFunction, priorityReduceFunction: priorityReduceFunction, args:                   args, </del> <ins> sharedLister:  handle.SnapshotSharedLister(), serviceLister: serviceLister, args:          args, </ins> }, nil } // ServiceAffinity is a plugin that checks service affinity. type ServiceAffinity struct { <del> args                   Args sharedLister           schedulerlisters.SharedLister serviceLister          corelisters.ServiceLister priorityMapFunction    priorities.PriorityMapFunction priorityReduceFunction priorities.PriorityReduceFunction </del> <ins> args          Args sharedLister  schedulerlisters.SharedLister serviceLister corelisters.ServiceLister </ins> } var _ framework.PreFilterPlugin = &ServiceAffinity{}

/sig scheduling /priority important-soon See as an example. /assign
/unassign /help
/assign
/remove-help


Question: framework.ExpectNoError(err, readmsg) } <ins> // testPodWithROVolume makes a minimal pod defining a volume input source. Similarly to // other tests for sig-windows this should append a nodeSelector for windows. </ins> func testPodWithROVolume(podName string, source v1.VolumeSource, path string) *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{

similar to PR en route
/sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: package azure import ( <del> "errors" </del> "fmt" "k8s.io/kubernetes/pkg/api/v1"

This is required to make Cloud Controller Manager operational on the various clouds that Kubernetes runs on. We have already PRs for the following clouds: GCE - AWS - Azure - OpenStack - The following clouds still need to be addresses: CloudStack Mesos OVirt Photon Rackspace vSphere If anyone is interested in contributing to this effort, please ping me directly (on kubernetes slack or through email)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle
cc:
I will take care of the vSphere and Photon.
I will take care of the Rackspace.
Merging for OpenStack
Reposting I am trying to use my custom out-of-tree cloud provider with Kubernetes. But I noticed that kubelet calls apis like with empty for cloud provider. Digging in the code for 1.7 branch, I see that is null for cloud-providers. As a result returns empty string. As a result, ByID() methods don't work for cloud providers since is not set.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: package service <del> import ( "strconv" "github.com/golang/glog" "k8s.io/kubernetes/pkg/api" ) </del> const ( // AnnotationLoadBalancerSourceRangesKey is the key of the annotation on a service to set allowed ingress ranges on their LoadBalancers //

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: testKubelet := newTestKubelet(t) kl := testKubelet.kubelet kl.nodeLister = testNodeLister{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}}, </del> <ins> { ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}, Status: api.NodeStatus{ Allocatable: api.ResourceList{ api.ResourcePods: *resource.NewQuantity(110, resource.DecimalSI), }, }, }, </ins> }} kl.nodeInfo = testNodeInfo{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}}, </del> <ins> { ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}, Status: api.NodeStatus{ Allocatable: api.ResourceList{ api.ResourcePods: *resource.NewQuantity(110, resource.DecimalSI), }, }, }, </ins> }} testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorapi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("DockerImagesFsInfo").Return(cadvisorapiv2.FsInfo{}, nil)

Was there some reason only the checking to the scheduler, not Kubelet?
IIRC it was written at the time when we stated adding checking in Kubelet. In other words - no reason.


Question: Factor:   6.0, Jitter:   0.1, } <del> err := wait.ExponentialBackoff(initBackoff, func() (bool, error) { klog.V(4).Infof("Initializing migrated drivers on CSINodeInfo") </del> <ins> err = wait.ExponentialBackoff(initBackoff, func() (bool, error) { klog.V(4).Infof("Initializing migrated drivers on CSINode") </ins> err := nim.InitializeCSINodeWithAnnotation() if err != nil { <del> kvh.SetKubeletError(fmt.Errorf("Failed to initialize CSINodeInfo: %v", err)) klog.Errorf("Failed to initialize CSINodeInfo: %v", err) </del> <ins> kvh.SetKubeletError(fmt.Errorf("Failed to initialize CSINode: %v", err)) klog.Errorf("Failed to initialize CSINode: %v", err) </ins> return false, nil }

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: import ( "context" <del> "fmt" </del> "strconv" "time"

I would like to propose the graduation of the following apiserver metrics from alpha to stable: With the exception of the metric these metrics have been stable for 2 years. The metric has been stable for 12 months.
/sig api-machinery /sig instrumentation
/assign /triage accepted
I'm okay with promoting all of these except I am hesitant about since it is relatively new. Can we omit that one?
I agree with the metric was patched a year ago as part of , so it would be best to wait a bit more until the majority of Kubernetes users catch up to the patched version and start using the new version of the metric in production.
A couple of renaming suggestions, since it is important to get it right before graduation: apiserverresponsesizes =apiserverresponsesizebytes aggregatorunavailableapiservice =aggregatorunavailableapiservices One concern regarding . It was very recently changed but the change was later reverted by In the end, the metric wasn't really changed, but it might raise the concern that some users may still need additional information from this metric so I am on the fence with this one. Side implementation note that we will need to keep in mind if we graduate is that the graduation should only affect the webhook admission metric and not the other, so we will need to pass the stability level to
Thank you and for the feedback. To clarify then, are the decisions to not graduate and final?
I'm actually okay with , the namespace addition was recent but it was a bad idea and was reverted. It has otherwise been in use for quite some time, so I'd be comfortable with promoting it.
I'm willing to make a PR for this one
+1 it wasn't a hard no from my side, just wanted to raise some concerns in case anyone had an objection.
/assign /unassign
Thank you for the feedback, and To summarize, we are in agreement to graduate all but the metric. Please correct me if I'm wrong.
/cc
since all the three metrics that are created with this function are graduating to stable, do you think I should still pass the to the function or should I just set the metrics created by this functions as stable by default? update: Seems like is designed to parse whatever's in front of . So putting a variable there (that is passed to the function) will cause the script to fail. Please correct me if I'm wrong.

update: Even setting the metrics to be stable by default will cause the script to fail and you'll get an error saying for the following line: It appears that the name of the stable metrics should be given to that object as a string and using the output of a function ( in this case) will cause the script to fail.
The PR is ready to be reviewed:


Question: } // makeEventRecorder sets up kubeDeps.Recorder if its nil. Its a no-op otherwise. <del> func makeEventRecorder(s *kubeletconfiginternal.KubeletConfiguration, kubeDeps *kubelet.Dependencies, nodeName types.NodeName) { </del> <ins> func makeEventRecorder(kubeDeps *kubelet.Dependencies, nodeName types.NodeName) { </ins> if kubeDeps.Recorder != nil { return }

/approve no-issue isn't working on , so I'm trying this instead. /cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
thanks


Question: apitypes "k8s.io/apimachinery/pkg/types" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubectl/pkg/util/podutils" <del> podutil "k8s.io/kubernetes/pkg/api/v1/pod" </del> "k8s.io/kubernetes/test/e2e/framework" testutils "k8s.io/kubernetes/test/utils" "k8s.io/kubernetes/test/utils/format"

Several test/e2e/framework/* packages have as their only dependency on k/k. The following functions are shared that way: IsPodAvailable returns true if a pod is available; false otherwise. IsPodReady returns true if a pod is ready; false otherwise. VisitContainers invokes the visitor function with a pointer to every container spec in the given pod spec with type set in mask. If visitor returns false, visiting is short-circuited. VisitContainers returns true if visiting completes, false if visiting was short-circuited. AllFeatureEnabledContainers returns a ContainerType mask which includes all container types except for the ones guarded by feature gate. The feature gate check in AllFeatureEnabledContainers is problematic. E2E tests don't know what the feature gates in the cluster are. The rest of the functions look like something that could be moved into == Having the functions in makes it clearer that the checks only depend on aspects of the public API. Reduces dependencies of e2e/framework on k/k, in case that we want to move it into staging. Prevents accidentally using checks that only work in-tree (like feature gates).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/assign


Question: return c.closeChan } <del> func (c *fakeConnection) RemoveStreams(_ ...httpstream.Stream) { </del> <ins> func (c *fakeConnection) RemoveStreams(streams ...httpstream.Stream) { for range streams { c.streamCount-- } </ins> } func (c *fakeConnection) SetIdleTimeout(timeout time.Duration) {

The port forwarding client leaks memory when a port forwarding session is reused over time. The port forwarder client should not leak memory. forwarding a port with kubectl. and close many connections to the forwarded port. The amount of memory leaked for each connection is quite small, so many connections are needed to observe the leak. This is related to , which was reported for the kubelet and fixed in . However, the same problem happens to the client. The client needs to call on the , otherwise the streams created by will be kept in memory until the port forwarder itself is closed, even if the streams themselves closed long before. <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
/cc


Question: Pid:                 eventPid, ProcessName:         processName, TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, } numExpectedOomEvents := len(oomInstancesToStream)

What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon


Question: "set_image.go", "set_resources.go", "set_selector.go", <ins> "set_serviceaccount.go", </ins> "set_subject.go", ], visibility = ["//build/visible_to:pkg_kubectl_cmd_set_CONSUMERS"],

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: goodConflictPlugins2.Profiles[0].Plugins.Score.Enabled = append(goodConflictPlugins2.Profiles[0].Plugins.Score.Enabled, config.Plugin{Name: "NodeResourcesMostAllocated", Weight: 2}) goodConflictPlugins2.Profiles[0].Plugins.Score.Enabled = append(goodConflictPlugins2.Profiles[0].Plugins.Score.Enabled, config.Plugin{Name: "RequestedToCapacityRatio", Weight: 2}) <ins> deprecatedPluginsConfig := validConfig.DeepCopy() deprecatedPluginsConfig.Profiles[0].PluginConfig = append(deprecatedPluginsConfig.Profiles[0].PluginConfig, config.PluginConfig{ Name: "NodeResourcesLeastAllocated", Args: &config.NodeResourcesLeastAllocatedArgs{}, }) </ins> scenarios := map[string]struct { expectedToFail bool config         *config.KubeSchedulerConfiguration

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Have a legacy scheduler config, upgrade to v1beta2 since we are at v1beta2 now, scheduler failed to start with some unreadable raw data, accurate and readable error message. config scheduler config like what has been said above. Kubernetes version (use ): latest. Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // // TODO: Any changes to a part of the object that represents desired state (labels, // annotations etc) should also increment the generation. <del> if !reflect.DeepEqual(oldDaemonSet.Spec.Template, newDaemonSet.Spec.Template) { </del> <ins> if !apiequality.Semantic.DeepEqual(oldDaemonSet.Spec.Template, newDaemonSet.Spec.Template) { </ins> newDaemonSet.Spec.TemplateGeneration = oldDaemonSet.Spec.TemplateGeneration + 1 newDaemonSet.Generation = oldDaemonSet.Generation + 1 return } <del> if !reflect.DeepEqual(oldDaemonSet.Spec, newDaemonSet.Spec) { </del> <ins> if !apiequality.Semantic.DeepEqual(oldDaemonSet.Spec, newDaemonSet.Spec) { </ins> newDaemonSet.Generation = oldDaemonSet.Generation + 1 } }

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: DigitalOcean OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others: What happened: when I do: it will run rolling update ONLY if there is a change. In case I do: Each time I run it - it will execute rolling update, which is misleading. What you expected to happen: I would like to have consistent behavior for Deployment and DaemonSet How to reproduce it (as minimally and precisely as possible): Create any DaemonSet with following:
Hmm, it shouldn't happen. TemplateGeneration is updated only when something changed. I will debug it.
I can reproduce this, and the DaemonSet is updated after every , even when nothing is changed. In , old/new DaemonSet template are seen as different with . I tried to change this part to use instead (needed to update code in validation as well), and the DaemonSet templates were then seen as the same, and thus the issue was gone. We used everywhere in the controllers. Is the right way to compare stuffs?
It's very strange. I tried to use pretty.Diff to print the differences but it returns empty slice. apiequality.Semantic.DeepEqual uses forked version of reflect.DeepEqual and it's little different from version in 1.7. will you prepare the patch?
The key of the reported DaemonSet config is its . Note that in the config file, (empty list); however, in the live DaemonSet object (created from the config file), (null list). That's why thinks they're different. If we use in the config file instead, won't change. From the field description, empty list and null list are different. Feel it's api machinery related
protobuf storage isn't able to store , so the object has that field set to when read back from storage. that is causing problems for that field in particular (where and have different meanings) see
cc for behavior of namespaces field
so it's not a DaemonSet bug
Well, still not sure whether Semantic.DeepEqual should be used or not. I'm still concerned with the behavior of the controller for other fields that can be populated with non-null empty lists by users, but serialize back out to null (or set omitempty)
deployments are using Maybe we should also switch?

If empty lists should be treated differently from null list, we should replace with in controller code.
that distinction is going away in all the fields I know of... I'm not super happy with Semantic.DeepEqual masking the difference between those two things, but that's probably the right thing for you to be using
Special meaning of [] was removed in I think a switch to Semantic.DeepEqual is still needed here
Filed


Question: currentInflightRequests.WithLabelValues(MutatingKind).Set(float64(mutating)) } <del> // Record records a single request to the standard metrics endpoints. For use by handlers that perform their own // processing. All API paths should use InstrumentRouteFunc implicitly. Use this instead of MonitorRequest if // you already have a RequestInfo object. func Record(req *http.Request, requestInfo *request.RequestInfo, component, contentType string, code int, responseSizeInBytes int, elapsed time.Duration) { </del> <ins> // RecordRequestTermination records that the request was terminated early as part of a resource // preservation or apiserver self-defense mechanism (e.g. timeouts, maxinflight throttling, // proxyHandler errors). RecordRequestTermination should only be called zero or one times // per request. func RecordRequestTermination(req *http.Request, requestInfo *request.RequestInfo, component string, code int) { </ins> if requestInfo == nil { requestInfo = &request.RequestInfo{Verb: req.Method, Path: req.URL.Path} }

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: .PHONY:	kbuild kpush <del> TAG = 1.21 </del> <ins> TAG = 1.25 </ins> # Rules for building the test image for deployment to Dockerhub with user kubernetes.

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: gvr("flowcontrol.apiserver.k8s.io", "v1beta3", "flowschemas"): { Stub:             `{"metadata": {"name": "fs-2"}, "spec": {"priorityLevelConfiguration": {"name": "name1"}}}`, ExpectedEtcdPath: "/registry/flowschemas/fs-2", <del> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta2", "FlowSchema"), </del> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: In one terminal, run `gcloud compute ssh master --ssh-flag="-L 8080:127.0.0.1:8080"` and in a second run `gcloud compute ssh master --ssh-flag="-R 8080:127.0.0.1:8080"`. <ins> ### OpenStack These instructions are for running on the command line.  Most of this you can also do through the Horizon dashboard. These instructions were tested on the Ice House release on a Metacloud distribution of OpenStack but should be similar if not the same across other versions/distributions of OpenStack. #### Make sure you can connect with OpenStack Make sure the environment variables are set for OpenStack such as: ```sh OS_TENANT_ID OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME ``` Test this works with something like: ``` nova list ``` #### Get a Suitable CoreOS Image You'll need a [suitable version of CoreOS image for OpenStack] (https://coreos.com/os/docs/latest/booting-on-openstack.html) Once you download that, upload it to glance.  An example is shown below: ```sh glance image-create --name CoreOS723  --container-format bare --disk-format qcow2  --file coreos_production_openstack_image.img  --is-public True ``` #### Create security group ```sh nova secgroup-create kubernetes "Kubernetes Security Group" nova secgroup-add-rule kubernetes tcp 22 22   0.0.0.0/0 nova secgroup-add-rule kubernetes tcp 80 80   0.0.0.0/0 ``` #### Provision the Master ```sh nova boot  --image <image_name>  --key-name <my_key>  --flavor <flavor id>  --security-group kubernetes  --user-data files/master.yaml  kube-master ``` ```<image_name>``` is the CoreOS image name.  In our example we can use the image we created in the previous step and put in 'CoreOS723' ```<my_key>``` is the keypair name that you already generated to access the instance. ```<flavor_id>``` is the flavor ID you use to size the instance.  Run ```nova flavor-list``` to get the IDs.  3 on the system this was tested with gives the m1.large size. The important part is to ensure you have the files/master.yml as this is what will do all the post boot configuration. This path is relevant so we are assuming in this example that you are running the nova command in a directory where there is a subdirectory called files that has the master.yml file in it.  Absolute paths also work. Next, assign it a public IP address: ``` nova floating-ip-list ``` Get an IP address that's free and run: ``` nova floating-ip-associate kube-master <ip address> ``` where ```<ip address>``` is the IP address that was available from the ```nova floating-ip-list``` command. #### Provision Worker Nodes Edit ```node.yaml``` and replace all instances of ```<master-private-ip>``` with the private IP address of the master node.  You can get this by runnning ```nova show kube-master``` assuming you named your instance kube master.  This is not the floating IP address you just assigned it. ```sh nova boot  --image <image_name>  --key-name <my_key>  --flavor <flavor id>  --security-group kubernetes  --user-data files/node.yaml  minion01 ``` This is basically the same as the master nodes but with the node.yaml post-boot script instead of the master. </ins> ### VMware Fusion #### Create the master config-drive

The file: has instructions for AWS, GCE, and VMware. I'd like to see documentation for OpenStack as well. I'd be happy to do the work for this as I don't think it would take too much time. I'll add some instructions to this issue if its something that people think would be helpful.
I would greatly appreciate any work you can do with this. I've been trying, unsuccessfully, to get the OpenStack integration working.
Ok, I created instructions in this gist: If it looks good to you I'll create a pull request and merge changes in the document after the GCE section (as it appears we're going in Alphabetical order? )
Sending a PR sounds good.


Question: return err } } <del> for _, externalAddr := range externalAddrs { if isClusterIPV4 != utilsnet.IsIPv4String(externalAddr) { framework.Logf("skipping testEndpointReachability() for external adddress %s as it does not match clusterIP (%s) family", externalAddr, clusterIP) continue } err := testEndpointReachability(externalAddr, sp.NodePort, sp.Protocol, pod) if err != nil { return err </del> <ins> if externalIPs { externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP) for _, externalAddr := range externalAddrs { if isClusterIPV4 != utilsnet.IsIPv4String(externalAddr) { framework.Logf("skipping testEndpointReachability() for external adddress %s as it does not match clusterIP (%s) family", externalAddr, clusterIP) continue } err := testEndpointReachability(externalAddr, sp.NodePort, sp.Protocol, pod) if err != nil { return err } </ins> } } return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: "authentication.go", "authn_audit.go", "authorization.go", <ins> "cachecontrol.go", </ins> "doc.go", "impersonation.go", "requestinfo.go",

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: # This tarball is only used by Ubuntu Trusty. KUBE_MANIFESTS_TAR= <del> if [[ "${OS_DISTRIBUTION}" == "trusty" ]]; then KUBE_MANIFESTS_TAR="${KUBE_ROOT}/server/kuernetes-manifests.tar.gz" </del> <ins> if [[ "${KUBE_OS_DISTRIBUTION}" == "trusty" ]]; then KUBE_MANIFESTS_TAR="${KUBE_ROOT}/server/kubernetes-manifests.tar.gz" </ins> if [[ ! -f "${KUBE_MANIFESTS_TAR}" ]]; then KUBE_MANIFESTS_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-manifests.tar.gz" fi

When I try to bring up my cluster (with a custom wrapper script that sources some other environment files,but calling in the end) using the AWS provider, the script fails with following reason: This happens because the correct environment variable set is . Changing the line to use this value fixes the issue.


Question: } status := *oldStatus.DeepCopy() for i := range status.ContainerStatuses { <del> if status.ContainerStatuses[i].State.Terminated != nil || status.ContainerStatuses[i].State.Waiting != nil { </del> <ins> if status.ContainerStatuses[i].State.Terminated != nil { </ins> continue } status.ContainerStatuses[i].State = v1.ContainerState{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped).

Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug.

Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .


Question: kubectl scale --replicas=3 statefulset/web`)) ) <del> const ( timeout = 5 * time.Minute ) </del> type ScaleOptions struct { FilenameOptions resource.FilenameOptions RecordFlags     *genericclioptions.RecordFlags

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: if I figure out --timeout,but the timeout is fixed in source not customized the timeout What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig cli


Question: prometheus.MustRegister(ContainersPerPodCount) prometheus.MustRegister(newPodAndContainerCollector(containerCache)) prometheus.MustRegister(PLEGRelistLatency) <ins> prometheus.MustRegister(PLEGDiscardEvents) </ins> prometheus.MustRegister(PLEGRelistInterval) prometheus.MustRegister(RuntimeOperations) prometheus.MustRegister(RuntimeOperationsLatency)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: return fmt.Errorf("waiting for controller-manager timed out") } <ins> // CheckForControllerManagerHealthy checks that the controller manager does not crash within "duration" func CheckForControllerManagerHealthy(duration time.Duration) error { var PID string cmd := "sudo docker ps | grep k8s_kube-controller-manager | cut -d ' ' -f 1" for start := time.Now(); time.Since(start) < duration; time.Sleep(5 * time.Second) { result, err := SSH(cmd, GetMasterHost()+":22", TestContext.Provider) if err != nil { // We don't necessarily know that it crashed, pipe could just be broken LogSSHResult(result) return fmt.Errorf("master unreachable after %v", time.Since(start)) } else if result.Code != 0 { LogSSHResult(result) return fmt.Errorf("SSH result code not 0. actually: %v after %v", result.Code, time.Since(start)) } else if result.Stdout != PID { if PID == "" { PID = result.Stdout } else { //its dead return fmt.Errorf("controller manager crashed, old PID: %s, new PID: %s", PID, result.Stdout) } } else { Logf("kube-controller-manager still healthy after %v", time.Since(start)) } } return nil } </ins> // Returns number of ready Nodes excluding Master Node. func NumberOfReadyNodes(c clientset.Interface) (int, error) { nodes, err := c.Core().Nodes().List(metav1.ListOptions{FieldSelector: fields.Set{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --We should have an e2e to test issue . The test needs to delete a PVC while a pod is still using it, and then restart the controller manager. /kind bug /sig storage
CC
/assign unofficially so I can find this later.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> watch "k8s.io/apimachinery/pkg/watch" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/scheme" <ins> watchtools "k8s.io/client-go/tools/watch" "k8s.io/client-go/util/retry" </ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" extensionsinternal "k8s.io/kubernetes/pkg/apis/extensions" "k8s.io/kubernetes/pkg/controller/daemon"

[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details