Question: Interval:       interval, Timeout:        timeout, CleanupPolicy:  updateCleanupPolicy, <del> MaxUnavailable: intstr.FromInt(0), </del> <ins> MaxUnavailable: intstr.FromInt(1), </ins> MaxSurge:       intstr.FromInt(1), } if rollback {

Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: all - whole test set OOMed Since when has it been failing: 2020-07-28 Testgrid link: https://k8s- Reason for failure: test pod OOMed after setting 16Gi memory limit Anything else we need to know:
/sig scalability /assign
/lgtm /approve
Not this window :) it was supposed to be for kubernetes/test-infra
Looks fixed?
Yeah - last run didn't OOM, so I think we can close this one.
Another 2 runs OOMed: /reopen
Reopened this issue. <detailsIn response to : Another 2 runs OOMed: </details
That seems really ridiculous to me. The fact that our performance tests can work with 16GB makes it super strange that simple correctness tests require 32GB+...
^^
/reopen Leaving discussion open
Reopened this issue. <detailsIn response to : /reopen Leaving discussion open </details
64Gi is too much, we need to find a proper limit: /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Testgrid is green

The default value of maxUnavailable seems to be 0 instead of 1. I tried rolling update for an RC with 3 replicas and it gave me: It should have said keep 2 pods available if maxUnavailable was 1. cc
It was falsely set as 0, I will submit a PR to help fix this.
Maybe we did this because before we introduced MaxUnavailable, rolling-update was written in a way that max unavailable was zero and we didnt want to change that behavior. might remember.
This changes fundamental behaviour in rolling-update. If I'm upgrading a single replica, rolling update now succeeds immediately, even though the new replica isn't ready. It even deletes the old RC before the new one is ready. With maxUnavailable set to 0, prior to this change, rolling-update would wait for the new replica to be ready before returning and deleting the old replica. I don't understand why this behaviour was a bug. Can you explain before I open an issue for this? (I expect this will hit a lot of people when 1.2 is released).
I think points out an interesting issue here: how should we handle the case where a RC with a single replica. Keep the default as 1(or 0) for all rolling-update, or set it as 0 when we are updating a RC with a single replica.
Yes I dont think that we want to change the behavior of kubectl rolling-update. I think it is better to revert to keeping the default for maxUnavailable as 0 for all kubectl rolling-update (as it was in 1.1) cc
maxSurge should be 1 and maxUnavailable should be 0 by default. We need to ensure that rollback works, however, in the case that the newRC is unavailable. cc


Question: "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <ins> "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", </ins> "//staging/src/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", <ins> "//staging/src/k8s.io/client-go/rest/fake:go_default_library", </ins> "//staging/src/k8s.io/client-go/testing:go_default_library", ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to use  in a unit test, however I get a nil pointer panic when is invoked. What you expected to happen: would return a that doesn't panic on any method call. Ideally, it would allow me to interact with the client in someway for testing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: client-go version: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
I believe this also impacts returning logs via a reactor as does not implement
Seems to be a problem with . Since we're creating the request with all empty fields. Not sure what the best way it to resolve this. The best thing I came up with is to use to return a request with everything set up. But I'm a noob here so that's just a shot in the dark :man_shrugging:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The request returned has a nil which is used in the
I have filed a PR trying to fix this issue.


Question: package azure import ( <ins> "bytes" </ins> "context" "encoding/json" "fmt"


Question: ginkgo.It("test Deployment ReplicaSet orphaning and adoption regarding controllerRef", func() { testDeploymentsControllerRef(f) }) <ins> ginkgo.It("Deployment should have a working scale subresource", func() { testDeploymentSubresources(f) }) </ins> /* Release: v1.12 Testname: Deployment Proportional Scaling

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. with this query you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a Deployment with a static label Read the Deployment Update the DeploymentScale Get the DeploymentScale to ensure it is updated Patch the DeploymentScale Read the DeploymentScale to ensure it is Patch Delete the Namespace and Deployment Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource readAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource replaceAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by N points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
Issue was presented in the Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints, and this issue was created for that purpose.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tests := []struct { args              []string p                 *execParams <ins> name              string </ins> expectError       bool expectedPod       string expectedContainer string <ins> expectedArgs      []string </ins> }{ { p:           &execParams{}, expectError: true, <ins> name:        "empty", </ins> }, { p:           &execParams{podName: "foo"}, expectError: true, <ins> name:        "no cmd", </ins> }, { p:           &execParams{podName: "foo", containerName: "bar"}, expectError: true, <ins> name:        "no cmd, w/ container", </ins> }, { <del> p:           &execParams{podName: "foo"}, args:        []string{"cmd"}, expectedPod: "foo", </del> <ins> p:            &execParams{podName: "foo"}, args:         []string{"cmd"}, expectedPod:  "foo", expectedArgs: []string{"cmd"}, name:         "pod in flags", </ins> }, { p:           &execParams{}, args:        []string{"foo"}, expectError: true, <ins> name:        "no cmd, w/o flags", </ins> }, { <del> p:           &execParams{}, args:        []string{"foo", "cmd"}, expectedPod: "foo", </del> <ins> p:            &execParams{}, args:         []string{"foo", "cmd"}, expectedPod:  "foo", expectedArgs: []string{"cmd"}, name:         "cmd, w/o flags", </ins> }, { p:                 &execParams{containerName: "bar"}, args:              []string{"foo", "cmd"}, expectedPod:       "foo", expectedContainer: "bar", <ins> expectedArgs:      []string{"cmd"}, name:              "cmd, container in flag", </ins> }, } for _, test := range tests { cmd := &cobra.Command{} <del> podName, containerName, err := extractPodAndContainer(cmd, test.args, test.p) </del> <ins> podName, containerName, args, err := extractPodAndContainer(cmd, test.args, test.p) </ins> if podName != test.expectedPod { <del> t.Errorf("expected: %s, got: %s", test.expectedPod, podName) </del> <ins> t.Errorf("expected: %s, got: %s (%s)", test.expectedPod, podName, test.name) </ins> } if containerName != test.expectedContainer { <del> t.Errorf("expected: %s, got: %s", test.expectedContainer, containerName) </del> <ins> t.Errorf("expected: %s, got: %s (%s)", test.expectedContainer, containerName, test.name) </ins> } if test.expectError && err == nil { <del> t.Error("unexpected non-error") </del> <ins> t.Errorf("unexpected non-error (%s)", test.name) </ins> } if !test.expectError && err != nil { <del> t.Errorf("unexpected error: %v", err) </del> <ins> t.Errorf("unexpected error: %v (%s)", err, test.name) } if !reflect.DeepEqual(test.expectedArgs, args) { t.Errorf("expected: %v, got %v (%s)", test.expectedArgs, args, test.name) </ins> } } }

When running , if you don't specify the deprecated and instead just do , it doesn't strip off the 1st command line arg, so sends the command to the container, whereas correctly sends just Regression from
FYI
We still support --, correct?
yes, we still support , but it's marked deprecated.
cc
has the fix. (and better unit test coverage) Sorry for the breakage!


Question: // makeNsenterArgs makes a list of argument to nsenter in order to do the // requested mount. <del> func makeNsenterArgs(source, target, fstype string, options []string) []string { </del> <ins> func (n *NsenterMounter) makeNsenterArgs(source, target, fstype string, options []string) []string { </ins> nsenterArgs := []string{ "--mount=/rootfs/proc/1/ns/mnt", <del> "/usr/bin/mount", </del> <ins> "--", n.absHostPath("mount"), </ins> } args := makeMountArgs(source, target, fstype, options)

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.


Question: err = r.(error) } }() <del> fs.Parse(strings.Split(c.argc, " ")) </del> <ins> fs.Parse(strings.Split(tc.argc, " ")) </ins> }() <del> if c.expectErr && err == nil { </del> <ins> if tc.expectErr && err == nil { </ins> t.Errorf("did not observe an expected error") continue } <del> if !c.expectErr && err != nil { t.Errorf("observed an unexpected error") </del> <ins> if !tc.expectErr && err != nil { t.Errorf("observed an unexpected error: %v", err) </ins> continue } <del> if c.expectVal != ip { t.Errorf("unexpected ip: expected %q, saw %q", c.expectVal, ip) </del> <ins> if tc.expectVal != ip { t.Errorf("unexpected ip: expected %q, saw %q", tc.expectVal, ip) } } } func TestIPPortVar(t *testing.T) { defaultIPPort := "0.0.0.0:8080" testCases := []struct { desc      string argc      string expectErr bool expectVal string }{ { desc:      "valid ipv4 1", argc:      "blah --ipport=0.0.0.0", expectVal: "0.0.0.0", }, { desc:      "valid ipv4 2", argc:      "blah --ipport=127.0.0.1", expectVal: "127.0.0.1", }, { desc:      "invalid IP", argc:      "blah --ipport=invalidip", expectErr: true, expectVal: defaultIPPort, }, { desc:      "valid ipv4 with port", argc:      "blah --ipport=0.0.0.0:8080", expectVal: "0.0.0.0:8080", }, { desc:      "invalid ipv4 with invalid port", argc:      "blah --ipport=0.0.0.0:invalidport", expectErr: true, expectVal: defaultIPPort, }, { desc:      "invalid IP with port", argc:      "blah --ipport=invalidip:8080", expectErr: true, expectVal: defaultIPPort, }, { desc:      "valid ipv6 1", argc:      "blah --ipport=::1", expectVal: "::1", }, { desc:      "valid ipv6 2", argc:      "blah --ipport=::", expectVal: "::", }, { desc:      "valid ipv6 with port", argc:      "blah --ipport=[::1]:8080", expectVal: "[::1]:8080", }, { desc:      "invalid ipv6 with port without bracket", argc:      "blah --ipport=fd00:f00d:600d:f00d:8080", expectErr: true, expectVal: defaultIPPort, }, } for _, tc := range testCases { fs := pflag.NewFlagSet("blah", pflag.PanicOnError) ipport := defaultIPPort fs.Var(IPPortVar{&ipport}, "ipport", "the ip:port") var err error func() { defer func() { if r := recover(); r != nil { err = r.(error) } }() fs.Parse(strings.Split(tc.argc, " ")) }() if tc.expectErr && err == nil { t.Errorf("%q: Did not observe an expected error", tc.desc) continue } if !tc.expectErr && err != nil { t.Errorf("%q: Observed an unexpected error: %v", tc.desc, err) continue } if tc.expectVal != ipport { t.Errorf("%q: Unexpected ipport: expected %q, saw %q", tc.desc, tc.expectVal, ipport) </ins> } } }

Whenever I try to change kube proxy metrics bind using flag , it gives me: Then, I the port , the error changes. I expect the metrics server to listen to 0.0.0.0:. Do I miss something? Environment: Kubernetes version: v1.7.8 Cloud provider or hardware configuration**: on-premise OS (e.g. from /etc/os-release): debian/stretch Kernel (e.g. ): Linux R11U2-7U20N-1 4.9.0-3-amd64 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy
The is being and does NOT allow specifying a port. Have you tried using a config file? The field allows you to specify a port (an example ).
/assign
Turned out we use a wrong type when validating and . Sent for fixing this. Will try to cherrypick the fix as well.
New finding, is broken on HEAD in a different way: Seems to be introduced by
Humm..We have some default logic for : Seems like for some reason it does not auto append the default port number OR the validation happens before defaulting.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
I got the same issue 0.0.0.0 0.0.0.0: both not work with kube-proxy 1.9.3 amd64
It is a BUG. Conflict between componentconfig.IPVar.Set(s string) and validation.validateHostPort.

Could you elaborate the issue a bit more? Which k8s version are you on?
I also meet the same issue with kubernetes 1.8. It's impossibe for Prometheus to fetch metrics from 127.0.0.1: in my case, any workaround? I think we'd better reopen this issue.
metrics binds to by default, are you trying to modify port as well? Thanks for reporting, I confirm the issue is remaining. (or any ip:port) does not work as intended even after the fix. Reopening this and filed a new issue for conflict kubeproxyconfig validation. /reopen
Thanks for reopening the issue. I want to bind metrics to 0.0.0.0: as well. With address 127.0.0.1:, Prometheus cannot pull metrics from this address unless they are on the same node.
is fixing this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale (I just got bitten by this on kube-proxy v1.9.10)
should fix this.
Closing this as is merged. /close
Closing this issue. <detailsIn response to : Closing this as is merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "io" "sync" <ins> "github.com/golang/glog" </ins> "k8s.io/apiserver/pkg/admission" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/cloudprovider"

In , we a Persistent Volume Controller with existing code that runs in cloud-controller-manager In , we cleaned up dependencies for AWS/GCE for the new controller Next steps, [x] Deprecate the PersistentVolumeLabel admission controller [ ] Remove the PersistentVolumeLabel admission controller once GCE and AWS can run as external cloud controller
/sig cluster-lifecycle
cc
/reopen
Changed the timeline for this to v1.10, sounds good as an initial plan?
/area platform/gce /sig gcp
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/unassign
adding kind/cleanup to this issue so that it can stay in 1.10. Please update the kind if that is incorrect. kind cleanup
/remove-lifecycle stale /lifecycle frozen is working on running the cloud controller in gce, but it's not there yet, so we shouldn't remove the PV label admission controller yet.
and should we drop the v1.10 milestone then for this issue?
reminder: this feature will be automatically dropped from 1.10 at Code Freeze tonight.
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing labels required for the v1.10 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
PVL admission controller was set for deprecation in 1.8, in 1.11 we'll be turning it off by default, users can still enable it if they need it which seems to be the use for GCE.
^ I'm not 100% aware of the status of PVLs in AWS/GCE, this is just the deprecation plan we had in place a while back. If there are any known issues with doing this please let me know
As long as the admission controller is not removed from cluster/gce/config-, then GCE should continue to work.

/remove-sig cluster-lifecycle
/assign /triage accepted
is it time to try this again?
I don't think we're ready. There's no replacement for it in the cloud controllers. The initial attempt at replacement using initializers was abandoned. I think we should consider a replacement using an admission webhook. cc
Could I suggest we make this the primary topic in an upcoming meeting? Maybe April 22nd Cloud Provider Extraction WG meeting? Assuming and can make it?
Yes that sounds good.
What do we think the way forward looks like?
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/remove-sig api-machinery Since sig cloud provider has taken care of this one. :)
do you know what the latest status is on this?
i saw this go by the other day, please see (search for )
xref:
!
For the record, replacement webhook:
Is this done now?
/close yes! deleted in
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr) } <del> waitForVolumeAttachedToNode(t, generatedVolumeName, nodeName2, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName2, cache.AttachStateAttached, asw) </del> <ins> waitForVolumeAttachStateToNode(t, generatedVolumeName, nodeName2, cache.AttachStateAttached, asw) </ins> } func Test_Run_UpdateNodeStatusFailBeforeOneVolumeDetachNodeWithReadWriteOnce(t *testing.T) {

pull-kubernetes-unit reconciler TestRunOneVolumeDetachFailNodeWithReadWriteOnce unknown, maybe recently? No response /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is still flaking but it is almost rare. Please see the link below for the test history. The root cause is that the detach operation is not triggered or completed within 100ms. 1) not triggered: 2) triggered but not completed: It seems no better way to fix it unless we increase sleep time in the test. i.e. (100ms -200ms) I'm not sure if it is valuable to fix it. cc what do you think?
any flake should be fixed even if it's rare.
/assign


Question: gvr("flowcontrol.apiserver.k8s.io", "v1alpha1", "flowschemas"): { Stub:             `{"metadata": {"name": "va1"}, "spec": {"priorityLevelConfiguration": {"name": "name1"}}}`, ExpectedEtcdPath: "/registry/flowschemas/va1", <del> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta2", "FlowSchema"), </del> <ins> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta3", "FlowSchema"), </ins> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: // VirtualMachinesClient defines needed functions for azure compute.VirtualMachinesClient type VirtualMachinesClient interface { CreateOrUpdate(ctx context.Context, resourceGroupName string, VMName string, parameters compute.VirtualMachine, source string) (resp *http.Response, err error) <ins> Update(ctx context.Context, resourceGroupName string, VMName string, parameters compute.VirtualMachineUpdate, source string) (resp *http.Response, err error) </ins> Get(ctx context.Context, resourceGroupName string, VMName string, expand compute.InstanceViewTypes) (result compute.VirtualMachine, err error) List(ctx context.Context, resourceGroupName string) (result []compute.VirtualMachine, err error) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: - linters: - ginkgolinter text: use a function call in (Eventually|Consistently) <ins> # SSA Extract calls are allowed in tests. - linters: - forbidigo text: should not be used because managedFields was removed path: _test.go$ </ins> linters: disable-all: false # in contrast to golangci.yaml, the default set of linters remains enabled enable: # please keep this alphabetized and in sync with golangci.yaml <ins> - forbidigo </ins> - ginkgolinter - gocritic - govet

Add a code static check for to report using server-side-apply extract With optimization of and , the will be removed in and , and maybe more of other components in the future, using Extract is problematic. It's safer to add this lint check.
/cc
/sig api-machinery
/sig scheduling /sig apps
/assign /triage accepted


Question: } } else { klog.V(2).Infof("azureDisk - attach disk(%s) succeeded", diskName) <del> // Invalidate the cache right after updating key := buildVmssCacheKey(nodeResourceGroup, ss.makeVmssVMName(ssName, instanceID)) ss.vmssVMCache.Delete(key) </del> } return err }

What happened: Azure Disks get randomly mounted in a way that makes them unusable, returning for every disk I/O operation. Affected Pods remain in until someone manually recreates them. What you expected to happen: Azure Disks are consistently usable by Pods inside which they are mounted. How to reproduce it: Create the following StatefulSet. Adjust the replicas and delete the associated Pods () until the problem occurs (i.e. some Pod ends up in ). <details</summary</details</brDefault StorageClass (as created by ACS-Engine): <details</summary</details</brAnything else we need to know?: Nodes were updated from 1.10.4 to 1.10.10, then from 1.10.10 to without a reboot. According to written by the issue is fixed, but I can still observe it quite often. Ref Azure/acs-engine Environment: Kubernetes version: v1.10.10 + Cloud provider or hardware configuration: Azure (westeurope) OS: Ubuntu 16.04.4 LTS Kernel: 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure
In my current test environment: (pod uid is -f23b-11e8-bf38-) on the node Notice the mismatch between the disk referenced by lun0 () and the actual mount (). on the node
After deleting the failing Pod and waiting for its successful recreation: (pod uid is now -f247-11e8-bf38-) on the node This time disks are matching (). on the node
/assign


Question: clientConfig.QPS = float32(s.KubeAPIQPS) clientConfig.Burst = int(s.KubeAPIBurst) <del> addChaosToClientConfig(s, clientConfig) </del> return clientConfig, nil } <del> // addChaosToClientConfig injects random errors into client connections if configured. func addChaosToClientConfig(s *options.KubeletServer, config *restclient.Config) { if s.ChaosChance != 0.0 { config.WrapTransport = func(rt http.RoundTripper) http.RoundTripper { seed := chaosclient.NewSeed(1) // TODO: introduce a standard chaos package with more tunables - this is just a proof of concept // TODO: introduce random latency and stalls return chaosclient.NewChaosRoundTripper(rt, chaosclient.LogChaos, seed.P(s.ChaosChance, chaosclient.ErrSimulatedConnectionResetByPeer)) } } } </del> // RunKubelet is responsible for setting up and running a kubelet.  It is used in three different applications: //   1 Integration tests //   2 Kubelet binary

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: } } <ins> // TestOptionsComplete checks that command line flags are combined with a // config properly. func TestOptionsComplete(t *testing.T) { header := `apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration ` // Determine default config (depends on platform defaults). o := NewOptions() require.NoError(t, o.Complete(new(pflag.FlagSet))) expected := o.config config := header + `logging: format: json flushFrequency: 1s verbosity: 10 vmodule: - filePattern: foo.go verbosity: 6 - filePattern: bar.go verbosity: 8 ` expectedLoggingConfig := logsapi.LoggingConfiguration{ Format:         "json", FlushFrequency: logsapi.TimeOrMetaDuration{Duration: metav1.Duration{Duration: time.Second}, SerializeAsString: true}, Verbosity:      10, VModule: []logsapi.VModuleItem{ { FilePattern: "foo.go", Verbosity:   6, }, { FilePattern: "bar.go", Verbosity:   8, }, }, Options: logsapi.FormatOptions{ JSON: logsapi.JSONOptions{ InfoBufferSize: resource.QuantityValue{Quantity: resource.MustParse("0")}, }, }, } for name, tc := range map[string]struct { config   string flags    []string expected *kubeproxyconfig.KubeProxyConfiguration }{ "empty": { expected: expected, }, "empty-config": { config:   header, expected: expected, }, "logging-config": { config: config, expected: func() *kubeproxyconfig.KubeProxyConfiguration { c := expected.DeepCopy() c.Logging = *expectedLoggingConfig.DeepCopy() return c }(), }, "flags": { flags: []string{ "-v=7", "--vmodule", "goo.go=8", }, expected: func() *kubeproxyconfig.KubeProxyConfiguration { c := expected.DeepCopy() c.Logging.Verbosity = 7 c.Logging.VModule = append(c.Logging.VModule, logsapi.VModuleItem{ FilePattern: "goo.go", Verbosity:   8, }) return c }(), }, "both": { config: config, flags: []string{ "-v=7", "--vmodule", "goo.go=8", "--ipvs-scheduler", "some-scheduler", // Overwritten by config. }, expected: func() *kubeproxyconfig.KubeProxyConfiguration { c := expected.DeepCopy() c.Logging = *expectedLoggingConfig.DeepCopy() // Flag wins. c.Logging.Verbosity = 7 // Flag and config get merged with command line flags first. c.Logging.VModule = append([]logsapi.VModuleItem{ { FilePattern: "goo.go", Verbosity:   8, }, }, c.Logging.VModule...) return c }(), }, } { t.Run(name, func(t *testing.T) { options := NewOptions() fs := new(pflag.FlagSet) options.AddFlags(fs) flags := tc.flags if len(tc.config) > 0 { tmp := t.TempDir() configFile := path.Join(tmp, "kube-proxy.conf") require.NoError(t, ioutil.WriteFile(configFile, []byte(tc.config), 0666)) flags = append(flags, "--config", configFile) } require.NoError(t, fs.Parse(flags)) require.NoError(t, options.Complete(fs)) assert.Equal(t, tc.expected, options.config) }) } } </ins> type fakeProxyServerLongRun struct{} // Run runs the specified ProxyServer.

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: Key | Value ------------- | ------------- <ins> `apiservers` | (Optional) The IP address / host name where a kubelet can get read-only access to kube-apiserver </ins> `cbr-cidr` | (Optional) The minion IP address range used for the docker container bridge. `cloud` | (Optional) Which IaaS platform is used to host kubernetes, *gce*, *azure*, *aws*, *vagrant* `cloud_provider` | (Optional) The cloud_provider used by apiserver: *gce*, *azure*, *vagrant*

The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver.

I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.


Question: } attachment, err := disk.waitForAttachmentStatus("attached") if err != nil { <ins> if err == wait.ErrWaitTimeout { c.applyUnSchedulableTaint(nodeName, "Volume stuck in attaching state - node needs reboot to fix impaired state.") } </ins> return "", err }

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: { name: "small mask IPv4", cidr: "192.168.1.1/28", <ins> want: 0, }, { name: "small mask IPv4", cidr: "192.168.1.1/27", </ins> want: 16, }, { name: "small mask IPv6", <ins> cidr: "fd00::1/124", want: 0, }, { name: "small mask IPv6", </ins> cidr: "fd00::1/122", want: 16, },

kube-apiserver observed a panic: kube-apiserver should not panic Not sure yet -- will add more details as I investigate. No response v1.25.4 N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /cc
max is greater than offset?
/assign


Question: {% set address = "-address=127.0.0.1" %} <ins> {% set publicAddressOverride = "" %} {% if grains.publicAddressOverride is defined %} {% set publicAddressOverride = "-public_address_override=" + grains.publicAddressOverride %} {% endif %} </ins> {% if grains.etcd_servers is defined %} {% set etcd_servers = "-etcd_servers=http://" + grains.etcd_servers + ":4001" %} {% else %}

The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver.

I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.


Question: "k8s.io/utils/clock" flowcontrol "k8s.io/api/flowcontrol/v1beta2" <del> flowcontrolapplyconfiguration "k8s.io/client-go/applyconfigurations/flowcontrol/v1beta2" </del> flowcontrolclient "k8s.io/client-go/kubernetes/typed/flowcontrol/v1beta2" flowcontrollister "k8s.io/client-go/listers/flowcontrol/v1beta2" )

A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted


Question: if err == nil { t.Fatal("expected error with json: prefix, got no error") } <del> if !strings.HasPrefix(err.Error(), "json: line 3:") { t.Fatalf("expected %q to have 'json: line 3:' prefix", err.Error()) </del> <ins> const msg = `json: offset 28: invalid character '"' after object key:value pair` if msg != err.Error() { t.Fatalf("expected %q, got %q", msg, err.Error()) </ins> } }

This bug was privately disclosed and told to open here. What happened: NewYAMLOrJSONDecoder from has an index out of bounds issue which can cause callers to panic. Download attached reproducer file Run golang test: When run with v0.19.3: Additionally, this can be seen in kubectl directly (below is kubectl 1.20.0): Environment: Kubernetes version (use ): Verified with 1.19, 1.20
/sig api-machinery
This piece attempts to find line+column position of a json syntax error. To do that it assumes that full input is available which is not true: is : is only the first data chunk: The subject happens e.g. when syntax error offset is past buffer size and Test to reproduce the panic: I am not sure it is easily possible to find line+column of json syntax error as full input may not be available. Maybe it should only attempt to find line+column if offset is within the first chunk (i.e. below buffer size).
Line number calculation in and updated in
do you want to submit a patch to with a defensive check to avoid the panic ( compare and )
I have raised a discussion on having line number in the . The consensus there is that it should be possible to count lines with a full input at hand and it does not worth to have it in the core library due to performance impact and maintenance costs. The problem of here is that it attempts to find out line number without having full input at hand. I think the defensive patch is not the right approach because line calculation is incorrect anyway even when there is no panic: begins at a value with a syntax error, i.e. in a stream of two values and syntax error in the second value it would contain only the second value and thus new lines in the first value in the stream would not be counted. I think the following options are there: to stream taken from . This is straightforward but changes interface. I can not find any code that uses field, it is only used in the error message that would change: line number only if offset is within the length of the first data chunk (). I am not sure what would be the unknown line number then, perhaps. line number properly during stream decoding, see This has some performance penalty as line number is calculated in any case even if there is no syntax error, i.e.

in the majority of the cases.
cc /triage accepted
This is probably acceptable.
I have opened a PR
Thank you for support.


Question: // Annotation sets a {k,v} pair to the inner pod annotation. func (p *PodWrapper) Annotation(key, value string) *PodWrapper { <del> if p.ObjectMeta.Annotations == nil { p.ObjectMeta.Annotations = make(map[string]string) } p.ObjectMeta.Annotations[key] = value </del> <ins> metav1.SetMetaDataAnnotation(&p.ObjectMeta, key, value) </ins> return p }

We need an integration test for the scheduler enforcement of the alpha feature. This is required for beta graduation; we need to collect signal on the current behavior before we can proceed. See this comment for context:
/sig storage /sig scheduler
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig storage /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: if err != nil { glog.V(1).Infof("%s probe for %q errored: %v", probeType, ctrName, err) if hasRef { <del> pb.recorder.Eventf(ref, v1.EventTypeWarning, events.ContainerUnhealthy, "%s probe errored: %v", probeType, err) </del> <ins> pb.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeWarning, events.ContainerUnhealthy, "%s probe errored: %v", probeType, err) </ins> } } else { // result != probe.Success glog.V(1).Infof("%s probe for %q failed (%v): %s", probeType, ctrName, result, output) if hasRef { <del> pb.recorder.Eventf(ref, v1.EventTypeWarning, events.ContainerUnhealthy, "%s probe failed: %s", probeType, output) </del> <ins> pb.recorder.Eventf(events.ToObjectReference(ref), v1.EventTypeWarning, events.ContainerUnhealthy, "%s probe failed: %s", probeType, output) </ins> } } return results.Failure, err

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): kubernetes/master using CRI with docker What happened: Create a pod that uses a readiness probe that does an exec to command that doesnt exist See the following in logs: What you expected to happen: No error in log, expected the event to properly be thrown. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know: Nope.
this is a regression.
debugging, will post a fix once found.
Other reports: ,
-- thanks for the other reports. at least now we have a reliable reproduction.
The problem appears to be the following: has a that correlates containers to a . It uses the type . event recorder library we use in the prober works with . If it doesn't get one of those, it expects the passed in object to implement List. As a result, we are passing a v1.ObjectReference the recorder was not expecting.
The bug may have been introduced in , which was there since v1.6.0.
Opened for a fix.
Keep this open until 1.6 is patched.
/remove-priority P0 /priority critical-urgent (I'm not actually sure this is the right priority, just trying to remove the old priority/PN labels)
Closing since has been merged.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 6 tree at Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu xenial Kernel (e.g. ): Linux fab7-compute-2 4.4.0-70-generic -Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm What happened: This gets logged ] Could not construct reference to: '&v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns--72wvm", UID:"-1edc-11e7-ac00-", APIVersion:"v1", ResourceVersion:"554", FieldPath:"spec.containers{kubedns}"}' due to: 'object does not implement the List interfaces'. Will not report event: 'Warning' 'Unhealthy' 'Readiness probe failed: Get http://10.1.0.130:8081/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)' What you expected to happen: Should log correctly How to reproduce it (as minimally and precisely as possible): Create pod that fails readiness probe
cc not sure if this is related to your changes
I don't think so. I didn't change the identification of or which are being used there.

I always get the same message. What does this exactly mean?
We're also seeing this on 1.6.1
Seeing this in as well
How we can come to know which release this is fixed? I am using 1.6.6 still I can see this issue. Thanks in Advance.


Question: // recommended as one of the prime benefits of exec plugins is that no secrets need // to be stored directly in the kubeconfig. // +k8s:conversion-gen=false <del> Config runtime.Object </del> <ins> Config runtime.Object `json:"-"` </ins> // InteractiveMode determines this plugin's relationship with standard input. Valid // values are "Never" (this exec plugin never uses standard input), "IfAvailable" (this

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: "fmt" "time" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/pkg/api/v1"

The merge queue is green with RBAC enabled, and the replicaset controller was missing permission to patch pods until fixed in Pod adoption and release depend on patching pods, which means that there are no merge-queue-blocking tests that test these features. Given that they are complex and rarely tested manually, we need tests that run as part of the merge queue to ensure they function correctly.
I'd like to work on this, where should the tests be ? I see that there are some patch-related tests for the replicaset controller in , and, according to http://submit-, the integration tests are run as part of the merge queue, is that correct? Thanks!
I think we'd want an e2e test that drives behavior externally on a normal cluster (by creating a replicaset, waiting to observe expected pods, modifies replicaset, waits to observe expected results, etc) . many of the integration tests tend to start up fake servers that don't share much in common with real deployments.
Ok thanks, on it.
cc:


Question: } func (eic *execInContainer) SetEnv(env []string) { <del> //unimplemented </del> <ins> // unimplemented </ins> } func (eic *execInContainer) Stop() { <del> //unimplemented </del> <ins> // unimplemented </ins> } func (eic *execInContainer) Start() error {

I was trying a scenario where liveness/readiness probe command run inside the container for long time('sleep 10 && exit 10') with shorter timeoutSecod value say 2 Sec. As expected container restarted because of timeout but when i check the events in pod describe it stated only unhealthy and in events actual reason for probe failure is not printed "liveness probe failed:<'no reason printed'". Without clear logs or pod events description it is very difficult to debug and understand what is the failure. Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " ! It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " Below is the example manifest file to reproduce the issue.

apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 Create a pod using above manifest file and do pod describe (kubectl describe pod <pod-name ) and check for the events. ! No response kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:59:53Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:52:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} openstack <details</details<details</detailscontainerd No response
I tried to fix one similar issue in
/sig node
/triage accepted
can I work on this issue? Thanks very much. /assign
We've experienced this issue with clusters on v1.20. The for that version mention the following change: Since v1.20 is seemingly when exec probe timeouts started being enforced, would it be possible to have this fix for this issue backported to all minor versions since then? Thanks!
bump on that question
What do you think of? Thanks very much.


Question: /var/cache/docker-install/{{ override_deb }}: file.managed: <del> - source: {{ storage_base }}{{ override_deb }} </del> <ins> - source: {{ override_deb_url }} </ins> - source_hash: sha1={{ override_deb_sha1 }} - user: root - group: root

GCE updated docker, but only via a base-image update. AWS should match versions of docker, to slipstream off all the testing that GCE gets. This will be slightly complicated because docker 1.8.3 isn't in the bucket AFAICT. Thanks to for the tip.
This is done by


Question: type listenPortOpener struct{} // OpenLocalPort holds the given local port open. <del> func (l *listenPortOpener) OpenLocalPort(lp *utilproxy.LocalPort) (utilproxy.Closeable, error) { return openLocalPort(lp) </del> <ins> func (l *listenPortOpener) OpenLocalPort(lp *utilproxy.LocalPort, isIPv6 bool) (utilproxy.Closeable, error) { return openLocalPort(lp, isIPv6) </ins> } // Proxier implements proxy.Provider

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: } const ( <ins> // maxPatchRetry is the maximum number of conflicts retry for during a patch operation before returning failure maxPatchRetry = 5 // backOffPeriod is the period to back off when apply patch resutls in error. backOffPeriod = 1 * time.Second // how many times we can retry before back off triesBeforeBackOff = 1 ) const ( </ins> apply_long = `Apply a configuration to a resource by filename or stdin. The resource will be created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create --save-config'.

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: spec: containers: - name: netexec <del> image: gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0 </del> <ins> image: gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.1 </ins> ports: - containerPort: 8080 protocol: TCP

/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign


Question: {" 100-200 ", true, "100-200", 200, 201}, {"0-0", true, "0-0", 0, 1}, {"", true, "", -1, 0}, <del> {"100", false, "", -1, -1}, </del> <ins> {"100", true, "100-100", 100, 101}, </ins> {"100 - 200", false, "", -1, -1}, {"-100", false, "", -1, -1}, {"100-", false, "", -1, -1}, {"200-100", false, "", -1, -1}, {"60000-70000", false, "", -1, -1}, {"70000-80000", false, "", -1, -1}, <ins> {"70000+80000", false, "", -1, -1}, {"1+0", true, "1-1", 1, 2}, {"0+0", true, "0-0", 0, 1}, {"1+-1", false, "", -1, -1}, {"1-+1", false, "", -1, -1}, {"100+200", true, "100-300", 300, 301}, {"1+65535", false, "", -1, -1}, {"0+65535", true, "0-65535", 65535, 65536}, </ins> } for i := range testCases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Implement this TODO: What you expected to happen: Start kube-proxy with a 'beginPort+offset' format port range configuration. This is needed when I want to allocate specific numbers of port from a host without extra manual calculation. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/sig network Does anyone besides kube-proxy use this?
/assign


Question: "//staging/src/k8s.io/api/policy/v1beta1:go_default_library", "//staging/src/k8s.io/api/rbac/v1:go_default_library", "//staging/src/k8s.io/api/rbac/v1beta1:go_default_library", <del> "//staging/src/k8s.io/api/scheduling/v1alpha1:go_default_library", </del> <ins> "//staging/src/k8s.io/api/scheduling/v1beta1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library",

/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling


Question: framework.ExpectNoError(results.ExitsBefore(init1, restartableInit1)) }) <del> ginkgo.It("should start first restartable init container before starting second init container", func() { framework.ExpectNoError(results.StartsBefore(restartableInit1, init2)) }) </del> ginkgo.It("should run first init container and first restartable init container together", func() { <del> framework.ExpectNoError(results.RunTogether(restartableInit1, init2)) </del> <ins> framework.ExpectNoError(results.RunTogetherLhsFirst(restartableInit1, init2)) </ins> }) ginkgo.It("should run second init container to completion before starting second restartable init container", func() {

master-blocking: ci-node-e2e Recent failures: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is also similar. The fix for that was bumping the delay time.
Hi Vyom - would this be a good first issue because there is a previous similar solution?
I hit the same failure in my . This seems caused because expects the regular container to start before the postStart: I posted a PR() to add to the postStart. However, there are other approaches such as: Introduce as mentioned . Remove from this test case because this check does not look the primary purpose of this testcase.
The test mentioned by is similar to this flake and doesn't use . I can just remove it. I'm also going to add a comment on RunTogether to indicate it expects ordering to be enforced. That being said, shouldn't PostStart be enforced to run after the container is started? /assign
Hey PRs are more than welcome :)
I've submitted the PR, let me know if I didn't follow any best practices or anything else is required, thanks!
Skipped after was merged. Last CI run passed. /close
Closing this issue. <detailsIn response to : Skipped after was merged. Last CI run passed. </details
/reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry.
Reopened this issue. <detailsIn response to : /reopen close the wrong one: is closed for swap test skip. this is for another flake. Sorry. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Well known tags TAG_KEY_MASTER_IP="kubernetes.io/master-ip" <ins> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION} </ins> # Defaults: ubuntu -> wily <del> if [[ "${KUBE_OS_DISTRIBUTION}" == "ubuntu" ]]; then KUBE_OS_DISTRIBUTION=wily </del> <ins> if [[ "${OS_DISTRIBUTION}" == "ubuntu" ]]; then OS_DISTRIBUTION=wily </ins> fi <del> # For GCE script compatibility OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION} case "${KUBE_OS_DISTRIBUTION}" in trusty|wheezy|jessie|vivid|wily|coreos) source "${KUBE_ROOT}/cluster/aws/${KUBE_OS_DISTRIBUTION}/util.sh" </del> <ins> # Loads the distro-specific utils script. # If the distro is not recommended, prints warnings or exits. function load_distro_utils () { case "${OS_DISTRIBUTION}" in jessie) ;; wily) ;; vivid) echo "vivid is currently end-of-life and does not get updates." >&2 echo "Please consider using wily or jessie instead" >&2 echo "(will continue in 10 seconds)" >&2 sleep 10 ;; coreos) echo "coreos is no longer supported by kube-up; please use jessie instead" >&2 exit 2 ;; trusty) echo "trusty is no longer supported by kube-up; please use jessie or wily instead" >&2 exit 2 ;; wheezy) echo "wheezy is no longer supported by kube-up; please use jessie instead" >&2 exit 2 </ins> ;; *) <del> echo "Cannot start cluster using os distro: ${KUBE_OS_DISTRIBUTION}" >&2 </del> <ins> echo "Cannot start cluster using os distro: ${OS_DISTRIBUTION}" >&2 echo "The current recommended distro is jessie" >&2 </ins> exit 2 ;; esac <ins> source "${KUBE_ROOT}/cluster/aws/${OS_DISTRIBUTION}/util.sh" } load_distro_utils </ins> # This removes the final character in bash (somehow) AWS_REGION=${ZONE%?}

If e.g. coreos doesn't work with kube-up, we should point users to an installation method that works. Also consider offering hints like "vivid no longer gets security updates, consider using wily or official"


Question: // Related issue #15262/PR #15263 to move apiserver to RFC339NANO is closed. func normalizeStatus(pod *v1.Pod, status *v1.PodStatus) *v1.PodStatus { bytesPerStatus := kubecontainer.MaxPodTerminationMessageLogLength <del> if containers := len(pod.Spec.Containers) + len(pod.Spec.InitContainers); containers > 0 { </del> <ins> if containers := len(pod.Spec.Containers) + len(pod.Spec.InitContainers) + len(pod.Spec.EphemeralContainers); containers > 0 { </ins> bytesPerStatus = bytesPerStatus / containers } normalizeTimeStamp := func(t *metav1.Time) {

is not normalized in in the status manager: This does not seem to cause any user facing problem because is sorted before passed to the status manager and timestamps are normalized at marshaling to create a patch: However, an unexpected behavior is caused internally because timestamps are not normalized. If the log verbosity is 3 or larger, the following messages are logged periodically in kubelet when an ephemeral container exists: This message is emitted here: After returns , is called. This does not look so harmful because gets eventually: API is called once unnecessarily in , though: It might be better to normalize ephemeral container statuses, at least timestamps, in the status manager to avoid this unexpected behavior. Set the log verbosity of kubelet to 3 or larger. Then, create an ephemeral container like . No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
This is a bug. Can you submit a PR to fix it? If not, I'd be happy to work on it
If you are interested, please feel free to assign yourself.
thx
/assign
/triage accepted /priority important-longterm


Question: # cleanup kubectl delete pods selector-test-pod <ins> # Create a deployment kubectl apply -f hack/testdata/null-propagation/deployment-null.yml "${kube_flags[@]:?}" # resources.limits.cpu should be nil. kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].resources.requests.cpu}" '' kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].resources.requests.memory}" '64Mi' # The default value of the terminationMessagePolicy field is `File`, so the result will not be changed. kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].terminationMessagePolicy}" 'File' # kubectl apply on create should do what kubectl apply on update will accomplish. kubectl apply -f hack/testdata/null-propagation/deployment-null.yml "${kube_flags[@]}" kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].resources.requests.cpu}" '' kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].resources.requests.memory}" '64Mi' kube::test::get_object_jsonpath_assert "deployment/my-dep-null" "{.spec.template.spec.containers[0].terminationMessagePolicy}" 'File' # hard.limits.cpu should be nil. kubectl apply -f hack/testdata/null-propagation/resourcesquota-null.yml "${kube_flags[@]}" kube::test::get_object_jsonpath_assert  "resourcequota/my-rq" "{.spec.hard['limits.cpu']}" '' kube::test::get_object_jsonpath_assert  "resourcequota/my-rq" "{.spec.hard['limits.memory']}" '' # kubectl apply on create should do what kubectl apply on update will accomplish. kubectl apply -f hack/testdata/null-propagation/resourcesquota-null.yml "${kube_flags[@]}" kube::test::get_object_jsonpath_assert "resourcequota/my-rq" "{.spec.hard['limits.cpu']}" '' kube::test::get_object_jsonpath_assert  "resourcequota/my-rq" "{.spec.hard['limits.memory']}" '' # cleanup kubectl delete deployment my-dep-null kubectl delete resourcequota my-rq </ins> ## kubectl apply --dry-run=server # Pre-Condition: no POD exists kube::test::get_object_assert pods "{{range.items}}{{${id_field:?}}}:{{end}}" ''

As per , I'm trying to remove CPU limits. But the behaviour is different depending whether the ResourceQuota exists or not: I would expect to be idempotent, and for its behaviour to result in the same end configuration regardless of the initial configuration. a new namespace named this command: that the ResourceQuota is created and is set to it again that is removed each subsequent time it's run, as long as resourcequota/compute-resource exists resourcequota/compute-resource again that the ResourceQuota is created and is set to again, and so forth the exact same behaviour if is changed to No response <details</details<detailsOpenShift 4.13.12 </details<details</details<details</details<details</details<details</details
/sig node
Tried to replicate the issue and observed the same behaviour.
/remove-sig node /sig api-machinery /cc
What's the goal of applying here? What are you trying to achieve? I agree that this is not the right behavior though.
I'm trying to remove a previously set property. I cannot remove the ResourceQuota altogether because it contains other properties (which I left out of my example) that I do not wish to remove.
But 0 is the same as unset, isn't it?
No, if I set it to 0 then any deployment with set to higher than 0 will fail to create any pods because it will exceed the quota.
/triage accepted /cc
/assign
Hello, I'm having a similar issue. Here's my use case: We use Flux to deploy things on our cluster - in particular, we use the component and set chart values in . Some of the charts we deploy have default values we want to remove, so we want to set a value to Editing an HelmRelease with this kind of ajustments drops the null keys, so default values end up being setup again and generated manifests are invalid :( Let me know if you need anything else to keep this issue moving!
I think the correct behavior is to set the struct to the default value when or , because in the standard library does this as well, and I think we expect the result to be as similar as possible to . So our real issue here is that we didn't set the to its default value when patch.
Based on the above, I think this question is not related to the current issue.

Tracing it back to , maybe the current result is what we want. The explicit null values means that we want to remove it in patch. As mentioned , our behavior in create is also as expected. I'm not quite sure if we actually want to fix it, and how /cc I hope to get your advice as you are the implementer of
I think client-side on create should do what on update will accomplish, namely, explicit nulls in the manifest are treated as the removal of the attributes they are values of. Proof of concept opened at , I'd be happy for someone to pick that up, add test coverage, and work through any failing tests that change uncovers
Thank you, but I still have a question. If we solve the difference between create and patch behaviors in ApiServer on the client side, will third-party callers still have the same problem? Or should we not cover this scenario?
I would consider changing behavior of null in a create request server-side to be a compatibility break. values are defined for client-side apply to mean "remove the key associated with this value". I think fixing the client-side apply creation case to have the same semantic is the most narrowly scoped fix we can make while preserving compatible server-side behavior generally.
Okay, I got it, I'll follow up with


Question: cleanupPods(cs, t, pods) cleanupNodes(cs, t) <ins> waitForSchedulerCacheCleanup(context.scheduler, t) </ins> }) } }

While lookint into , I spent few minutes looking into this test: And there is a problem that is relatively easy to reproduce. The problem is that we are running its test cases one by one. But nothing in the meantime is ensuring that the current state that the test assumes is reflected in scheduler. In particular the problem that I observed couple time is that: we finished one test case we call cleanupNodes [but we don't ensure that scheduler actually cleaned up this node] we start the next test case we try to schedule a pod that has some toleration (but shouldn't fit on our node) by accident the toleration make it fit on to the previous node so scheduler schedules it on the node from previous test case
/assign


Question: } } <del> // Install the kubernetes-specific postrouting rules. We use a whole chain for // this so that it is easier to flush and change, for example if the mark // value should ever change. // NB: THIS MUST MATCH the corresponding code in the kubelet writeLine(proxier.natRules, []string{ "-A", string(kubePostroutingChain), "-m", "mark", "!", "--mark", fmt.Sprintf("%s/%s", proxier.masqueradeMark, proxier.masqueradeMark), "-j", "RETURN", }...) // Clear the mark to avoid re-masquerading if the packet re-traverses the network stack. writeLine(proxier.natRules, []string{ "-A", string(kubePostroutingChain), // XOR proxier.masqueradeMark to unset it "-j", "MARK", "--xor-mark", proxier.masqueradeMark, }...) masqRule := []string{ "-A", string(kubePostroutingChain), "-m", "comment", "--comment", `"kubernetes service traffic requiring SNAT"`, "-j", "MASQUERADE", } if proxier.iptables.HasRandomFully() { masqRule = append(masqRule, "--random-fully") } writeLine(proxier.natRules, masqRule...) // Install the kubernetes-specific masquerade mark rule. We use a whole chain for // this so that it is easier to flush and change, for example if the mark // value should ever change. writeLine(proxier.natRules, []string{ "-A", string(KubeMarkMasqChain), "-j", "MARK", "--or-mark", proxier.masqueradeMark, }...) </del> } // getExistingChains get iptables-save output so we can check for existing chains and rules.

Which jobs are failing: pull-kubernetes-e2e-gci-gce-ipvs Which test(s) are failing: Not all of these fail on every run: [sig-network] Networking Granular Checks: Services should function for endpoint-Service: http [sig-network] Services should allow pods to hairpin back to themselves through services [sig-storage] Flexvolumes should be mountable when non-attachable [sig-network] Networking Granular Checks: Services should function for endpoint-Service: udp [sig-network] Services should have session affinity timeout work for service with type clusterIP [LinuxOnly] [Conformance] [sig-network] Services should allow pods to hairpin back to themselves through services Since when has it been failing: Looking through job history, it looks like the last time any runs passed was on May 2. Testgrid link: N/A since testgrid filtered tests based on [sig-network] Anything else we need to know: This was discovered by in , it also appears to be affecting other PRs. /cc /sig network
This seems related to , but if I'm interpreting test results correctly pull-kubernetes-e2e-gci-gce-ipvs never stopped failing.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign I'll try to fix those cases
May I ask why do you think this is related to ? Seems like first two flexvolumes related cases failed, then after some commit, service related cases start failing.
let me handle the services tests, I think I know the problem, will try to submit a fix today. Do you know what happen with the storage ones?
/assign I just submitted a PR to fix the failing tests
Thanks, seems like all cases fixed.


Question: # First build a release $(dirname $0)/../release/release.sh <ins> if [ "$?" != "0" ]; then echo "Building the release failed!" exit 1 fi </ins> # Now bring a new cluster up with that release. $(dirname $0)/../cluster/kube-up.sh

dev-build-and-{push|up}.sh scripts silently push an old version.


Question: expectAssumedClaims = append(expectAssumedClaims, expected.assumedClaim) } actualAssumedClaims := tc.listAssumedClaims() <del> assert.Equal(t, expectAssumedClaims, actualAssumedClaims, "assumed claims") </del> <ins> if diff := cmp.Diff(expectAssumedClaims, actualAssumedClaims, cmpopts.IgnoreFields(metav1.ObjectMeta{}, "UID", "ResourceVersion")); diff != "" { t.Errorf("Assumed claims are different (- expected, + actual):n%s", diff) } </ins> var expectInFlightClaims []metav1.Object if expected.inFlightClaim != nil { expectInFlightClaims = append(expectInFlightClaims, expected.inFlightClaim) } actualInFlightClaims := tc.listInFlightClaims() <del> assert.Equal(t, expectInFlightClaims, actualInFlightClaims, "in-flight claims") } // setGVK is implemented by metav1.TypeMeta and thus all API objects, in // contrast to metav1.Type, which is not (?!) implemented. type setGVK interface { SetGroupVersionKind(gvk schema.GroupVersionKind) } // stripObjects removes certain fields (Kind, APIVersion, etc.) which are not // important and might not be set. func stripObjects(objects []metav1.Object) { for _, obj := range objects { obj.SetResourceVersion("") obj.SetUID("") if objType, ok := obj.(setGVK); ok { objType.SetGroupVersionKind(schema.GroupVersionKind{}) } </del> <ins> if diff := cmp.Diff(expectInFlightClaims, actualInFlightClaims, cmpopts.IgnoreFields(metav1.ObjectMeta{}, "UID", "ResourceVersion")); diff != "" { t.Errorf("In-flight claims are different (- expected, + actual):n%s", diff) </ins> } }

pull-kubernetes-unit dynamicresources unknown multiple failures in tests as seen in n/a /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably cc
/assign
DeepCopy missing here:


Question: return p } <ins> // Terminating sets the inner pod's deletionTimestamp to current timestamp. func (p *PodWrapper) Terminating() *PodWrapper { now := metav1.Now() p.DeletionTimestamp = &now return p } </ins> // ZeroTerminationGracePeriod sets the TerminationGracePeriodSeconds of the inner pod to zero. func (p *PodWrapper) ZeroTerminationGracePeriod() *PodWrapper { p.Spec.TerminationGracePeriodSeconds = &zero

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: newSCSICreated = true } <del> disk := vmDevices.CreateDisk(diskController, ds.Reference(), vmDiskPath) </del> <ins> disk := vmDevices.CreateDisk(scsiController, ds.Reference(), vmDiskPath) unitNumber, err := getNextUnitNumber(vmDevices, scsiController) if err != nil { glog.Errorf("cannot attach disk to VM, limit reached - %v.", err) return "", "", err } *disk.UnitNumber = unitNumber </ins> backing := disk.Backing.(*types.VirtualDiskFlatVer2BackingInfo) backing.DiskMode = string(types.VirtualDiskModeIndependent_persistent)

vSphere cloud provider volume plugin attach disk limits the number of volumes because of two issues: new SCSI controller is only created when one does not exist. A new controller should be when required and limit not reached. disk will fail when attaching disk to slot 7 on controller. Slot 7 on SCSI controller is reserved and therefore should be skipped. This issue has been brought up in Govmomi library .