Question: import ( "net" <ins> "reflect" </ins> "strings" "testing"

Consider a service like The API refuses to accept this service, stating It works fine without . It works when using different ports. The iptables rules installed (in mode) are TCP/UDP specific. I'm not sure if this is deliberate but not well documented, or a bug? I don't see a reason not to allow a UDP and TCP node port to have the same number.
For context, I am trying to make the cluster DNS available to legacy systems with a bit of forwarding, and DNS really really wants to have UDP and TCP on the same port number.
what is your kubernetes version?
This is on 1.1.4. From a quick reading of the code this hasn't changed though.
This was intentionally omitted due to laziness and complexity. We hoped nobody would need it, since NodePorts are generally allocated automatically. It's mostly going to be a matter of adding a switch to look at 2 different saved allocation maps and figuring out how to transition. , Matthias Rampke wrote:
I've also hit this issue (see: ) when trying to expose DNS to queries from outside the cluster. Note that in my setup, the "NodePort" is different for UDP and TCP but only UDP appears to work (it's the first to be defined?).
I also run into this issue. I've got a service that wants to expose TCP and UDP port with the same number. Any plans to include this issue in any roadmap?
Help wanted. We're buried in high-priority bugs and features. I ACK that this behaves imperfectly, but I and my team will not have time to look at it in the immediate future.
FYI
I think the root cause is , when serviceport allocator attempts to reserve the provided port, it just use the port number not consider the port protocol. This is a bug of service port allocator I believe.
"bug", but yes. We'd need to store parallel bitmaps per protocol. Even then, it's not obvious when to use the same port number in different protocols, and whether that is REQUIRED or not.

If you have a Service with port 53 in TCP and UDP, and I can't find any port number that is available in both protocols, should I: a) fail and reject the REST operation b) choose a different port number c) retry forever and hang the REST operation d) accept the REST operation but log an event saying I failed to do my job NodePort was really made for load-balancers, where the port number doesn't matter. Humans using it is "unexpected". , TonyAdo wrote:
this is what I'd expect. Port exhaustion is not really a temporary Matthias Rampke Engineer SoundCloud Ltd. +49 173 Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. HRB 110657B
I am afraid PR will not fix this issue. We have to store parallel bitmaps per protocol. I will open a new PR to fix this issue.
This alone is insufficient, but we might get away without storing two maps by simply saying the port map is unified across protocols if a service has 2 ports that use the same port number but different protocols, allocate the first nodePort and use the same value for the 2nd port This will prevent the case of ports being used in one protocol but not the other and failing to allocate. Something like: I haven;t tried too hard to break it, but something like that MIGHT work.
Additionally, if you have to store multiple bitmaps, you have a serious hassle on your hands for upgrading cluster in-place. Better to avoid it if you can.
sure, I will think about this more.

Hi, I have created and deployed a service similar to below in minikube environment.. however I am not to connect to the ports sometimes TCP/UDP ` apiVersion: v1 kind: Service metadata: name: dns spec: selector: … clusterIP: … type: NodePort ports: name: grpc protocol: TCP port: targetPort: grpc-port name: port-udp port: targetPort: 5062 protocol: UDP nodePort: name: port-tcp port: targetPort: 5062 protocol: TCP nodePort: Example: $kubectl get all service/-service NodePort 10.106.159.94 <none50051:/TCP,30:/UDP,:/TCP $ telnet 192.168.99.100 telnet: can't connect to remote host (192.168.99.100): Connection refused $ telnet localhost telnet: can't connect to remote host (127.0.0.1): Connection refused $nc -z -v -u 192.168.99.100 Connection to 192.168.99.100 port [udp/] succeeded! $ nc -z -v 192.168.99.100 nc: connect to 192.168.99.100 port (tcp) failed: Connection refused $ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} $netstat -an tcp 0 0 ::: ::: LISTEN udp 0 0 ::: :::*
!


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package state import ( "os" "strings" "testing" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager" testutil "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state/testing" ) const testingCheckpoint = "dramanager_checkpoint_test" // TODO (https://github.com/kubernetes/kubernetes/issues/123552): reconsider what data gets stored in checkpoints and whether that is really necessary. func TestCheckpointGetOrCreate(t *testing.T) { testCases := []struct { description                string checkpointContent          string expectedError              string expectedClaimInfoStateList ClaimInfoStateList }{ { description:                "new-checkpoint", expectedClaimInfoStateList: nil, }, { description:       "single-claim-info-state", checkpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}}]}","Checksum":1656016162}`, expectedClaimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, }, { description:       "claim-info-state-with-multiple-devices", checkpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]},{"PoolName":"worker-1","DeviceName":"dev-2","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}}]}","Checksum":3369508096}`, expectedClaimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, { PoolName:     "worker-1", DeviceName:   "dev-2", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, }, { description:       "two-claim-info-states", checkpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example-1","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}},{"ClaimUID":"4cf8db2d-06c0-7d70-1a51-e59b25b2c16c","ClaimName":"example-2","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-2","RequestNames":null,"CDIDeviceIDs":null}]}}}]}","Checksum":1582256999}`, expectedClaimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example-1", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:   "worker-1", DeviceName: "dev-2", }, }, }, }, ClaimUID:  "4cf8db2d-06c0-7d70-1a51-e59b25b2c16c", ClaimName: "example-2", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, }, { description:       "incorrect-checksum", checkpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","Entries":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example-1","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}},{"ClaimUID":"4cf8db2d-06c0-7d70-1a51-e59b25b2c16c","ClaimName":"example-2","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-2","RequestNames":null,"CDIDeviceIDs":null}]}}}]}","Checksum":2930258365}`, expectedError:     "checkpoint is corrupted", }, { description:       "invalid-JSON", checkpointContent: `{`, expectedError:     "unexpected end of JSON input", }, } // create temp dir testingDir, err := os.MkdirTemp("", "dramanager_state_test") if err != nil { t.Fatal(err) } defer func() { if err := os.RemoveAll(testingDir); err != nil { t.Fatal(err) } }() // create checkpoint manager for testing cpm, err := checkpointmanager.NewCheckpointManager(testingDir) require.NoError(t, err, "could not create testing checkpoint manager") for _, tc := range testCases { t.Run(tc.description, func(t *testing.T) { // ensure there is no previous checkpoint require.NoError(t, cpm.RemoveCheckpoint(testingCheckpoint), "could not remove testing checkpoint") // prepare checkpoint for testing if strings.TrimSpace(tc.checkpointContent) != "" { mock := &testutil.MockCheckpoint{Content: tc.checkpointContent} require.NoError(t, cpm.CreateCheckpoint(testingCheckpoint, mock), "could not create testing checkpoint") } checkpointer, err := NewCheckpointer(testingDir, testingCheckpoint) require.NoError(t, err, "could not create testing checkpointer") checkpoint, err := checkpointer.GetOrCreate() if strings.TrimSpace(tc.expectedError) != "" { assert.ErrorContains(t, err, tc.expectedError) } else { require.NoError(t, err, "unexpected error") stateList, err := checkpoint.GetClaimInfoStateList() require.NoError(t, err, "could not get data entries from checkpoint") require.NoError(t, err) assert.Equal(t, tc.expectedClaimInfoStateList, stateList) } }) } } func TestCheckpointStateStore(t *testing.T) { testCases := []struct { description               string claimInfoStateList        ClaimInfoStateList expectedCheckpointContent string }{ { description: "single-claim-info-state", claimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, expectedCheckpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}}]}","Checksum":1656016162}`, }, { description: "claim-info-state-with-multiple-devices", claimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, { PoolName:     "worker-1", DeviceName:   "dev-2", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, expectedCheckpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]},{"PoolName":"worker-1","DeviceName":"dev-2","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}}]}","Checksum":3369508096}`, }, { description: "two-claim-info-states", claimInfoStateList: ClaimInfoStateList{ { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:     "worker-1", DeviceName:   "dev-1", RequestNames: []string{"test request"}, CDIDeviceIDs: []string{"example.com/example=cdi-example"}, }, }, }, }, ClaimUID:  "067798be-454e-4be4-9047-1aa06aea63f7", ClaimName: "example-1", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, { DriverState: map[string]DriverState{ "test-driver.cdi.k8s.io": { Devices: []Device{ { PoolName:   "worker-1", DeviceName: "dev-2", }, }, }, }, ClaimUID:  "4cf8db2d-06c0-7d70-1a51-e59b25b2c16c", ClaimName: "example-2", Namespace: "default", PodUIDs:   sets.New("139cdb46-f989-4f17-9561-ca10cfb509a6"), }, }, expectedCheckpointContent: `{"Data":"{"kind":"DRACheckpoint","apiVersion":"checkpoint.dra.kubelet.k8s.io/v1","ClaimInfoStateList":[{"ClaimUID":"067798be-454e-4be4-9047-1aa06aea63f7","ClaimName":"example-1","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-1","RequestNames":["test request"],"CDIDeviceIDs":["example.com/example=cdi-example"]}]}}},{"ClaimUID":"4cf8db2d-06c0-7d70-1a51-e59b25b2c16c","ClaimName":"example-2","Namespace":"default","PodUIDs":{"139cdb46-f989-4f17-9561-ca10cfb509a6":{}},"DriverState":{"test-driver.cdi.k8s.io":{"Devices":[{"PoolName":"worker-1","DeviceName":"dev-2","RequestNames":null,"CDIDeviceIDs":null}]}}}]}","Checksum":1582256999}`, }, } // Should return an error, stateDir cannot be an empty string if _, err := NewCheckpointer("", testingCheckpoint); err == nil { t.Fatal("expected error but got nil") } // create temp dir testingDir, err := os.MkdirTemp("", "dramanager_state_test") if err != nil { t.Fatal(err) } defer func() { if err := os.RemoveAll(testingDir); err != nil { t.Fatal(err) } }() // NewCheckpointState with an empty checkpointName should return an error if _, err = NewCheckpointer(testingDir, ""); err == nil { t.Fatal("expected error but got nil") } cpm, err := checkpointmanager.NewCheckpointManager(testingDir) require.NoError(t, err, "could not create testing checkpoint manager") require.NoError(t, cpm.RemoveCheckpoint(testingCheckpoint), "could not remove testing checkpoint") cs, err := NewCheckpointer(testingDir, testingCheckpoint) require.NoError(t, err, "could not create testing checkpointState instance") for _, tc := range testCases { t.Run(tc.description, func(t *testing.T) { checkpoint, err := NewCheckpoint(tc.claimInfoStateList) require.NoError(t, err, "could not create Checkpoint") err = cs.Store(checkpoint) require.NoError(t, err, "could not store checkpoint") err = cpm.GetCheckpoint(testingCheckpoint, checkpoint) require.NoError(t, err, "could not get checkpoint") checkpointContent, err := checkpoint.MarshalCheckpoint() require.NoError(t, err, "could not Marshal Checkpoint") assert.Equal(t, tc.expectedCheckpointContent, string(checkpointContent)) }) } } </ins>

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: // nodeMontiorGracePeriod is the amount of time which we allow a running node to be // unresponsive before marking it unhealthy. Must be N times more than kubelet's // nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet <del> // to post node status. </del> <ins> // to post node status. This value should also be greater than the sum of // HTTP2_PING_TIMEOUT_SECONDS and HTTP2_READ_IDLE_TIMEOUT_SECONDS. </ins> NodeMonitorGracePeriod metav1.Duration // podEvictionTimeout is the grace period for deleting pods on failed nodes. PodEvictionTimeout metav1.Duration

The controller-manager node-lifecycle-controller watches node lease and update probeTimestamp , and it check node probeTimestamp is before . The node-monitor-grace-period default value is 40s. When the client connection lost, the lease watcher cannot receive new event. With the http2 health check, HTTP2PINGTIMEOUTSECONDS default is 30s, HTTP2READIDLETIMEOUTSECONDS default is 15s, it will delay 45s to kill http2 connection. During this period, node-lifecycle-controller will mark all nodes NotReady. Here are some logs When the http2 connection lost, the watch should stop as soon as possible. The reflector should relist resource, and node-lifecycle-controller should not mark all node NotReady. HTTP2PINGTIMEOUTSECONDS+HTTP2READIDLETIMEOUTSECONDS should be less than 40s. the controller-manager node-lifecycle-controller watch client srcIP:srcPort. Drop all tcp package send into srcIP:srcPort. for more than 40 seconds, all node will be marked NotReady node will be Ready after some seconds. No response <details</details<details</details<details</details<details</details<details</details<details</details
How about double-check node lease before marking node NotReady? The node-lifecycle-controller can get reference lease from apiserver with
/sig node
/sig api-machinery
/cc We have discussed this timeouts values before and myself, I also think got some metrics with lower timeouts for the HTTP2 connections. In this situation is clear that the defaults of the node watcher and the timeouts are related and having good defaults will be ideal, but the question is which values do we change ?
I think in internal network, the ping response almost return in 1 second, the HTTP2PINGTIMEOUT_SECONDS default value should be 5 seconds. And the total timeout will 35 seconds. WDYT?
/triage accepted any pointers in other places where those timeouts were changed for http2? Any notes from those discussions?
/priority important-longterm since it is not a regression I will keep it longterm priority and not higher. But we can reevaluate if we hear more reports about it.

/assign
I'm investigating problems similar to this, but I'm looking for a way to confirm this is what is happening - and if there are tips for how to recreate such a scenario, that would be very helpful. We get infrequent reports of "worker goes NotReady for a short time" incidents that appear to be triggered by events like failing over a LB sitting front of the kube-apiservers. We've done some tests with a simple client-go app and have recreated a scenario in which the app logs messages after 45 secs of "context exceeded" errors. The failures reported to us from customers don't have http2 errors in the kubelet logs, and our attempts to recreate with kubelet have never shown more more than a single error which works on subsequent retry.
Hi folks, we have a simulated recreate of the problem. This recreate involves injecting a 40 second delay in "connectivity" between the Kubernetes API server and etcd (we a 40 second delay to etcd defrag and defragged all etcd members at the same time). The connectivity between these two components is obviously critical to nodes successfully updating their leases in order to stay ready. We use 40 seconds because it matches . The test will usually cause one or more of the cluster nodes will flap their readiness state. Setting on the Kubernetes controller manager prevents the flap. Given this, I think that one or more of the Kubernetes default configurations needs to change in order to tolerate such "connectivity" problems in order to maintain healthy node state.
What's the proposal then? Do we update --node-monitor-grace-period to 50s? It seems the less risky change as the client-go defaults are more sensitive and impact the whole ecosystem, beyond kubernetes itself are you open to increase this default? It seems worth it to improve the reliability of cluster
I think we should update .
can someone send a PR to see how it will look like?
I have been following this discussion as we had encountered the bug as well. I have created a PR - Please let me know if it works
I'll add another data point from our testing. Namely, the simulated recreate noted depends on the Kubernetes controller manager (KCM) leader election configuration. Using the following leader election configuration for the KCM yields a KCM that has no leader after the delayed etcd defrag.

As a result, nodes won't be marked not ready unless the KCM leadership is restored before the nodes renew their lease. Extending the KCM leader election configuration to the following, makes the problem easier to recreate.
/assign Reassigning the issue to me


Question: reviewers: - sig-instrumentation-reviewers labels: <del> - sig/gcp </del> <ins> - area/provider/gcp </ins> - sig/instrumentation

Some owners files attempt to auto apply the no longer existing label, which is now : This is present in 4 owners files: /assign /area oncall-hotlist
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: } serviceEventHandler = proxierUserspace proxier = proxierUserspace <del> // Remove artifacts from the iptables and ipvs Proxier, if not on Windows. klog.V(0).Info("Tearing down inactive rules.") // TODO this has side effects that should only happen when Run() is invoked. iptables.CleanupLeftovers(iptInterface) // IPVS Proxier will generate some iptables rules, need to clean them before switching to other proxy mode. // Besides, ipvs proxier will create some ipvs rules as well.  Because there is no way to tell if a given // ipvs rule is created by IPVS proxier or not.  Users should explicitly specify `--clean-ipvs=true` to flush // all ipvs rules when kube-proxy start up.  Users do this operation should be with caution. if canUseIPVS { ipvs.CleanupLeftovers(ipvsInterface, iptInterface, ipsetInterface, cleanupIPVS) } </del> } iptInterface.AddReloadFunc(proxier.Sync)

What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

What happened: When restarting kube-proxy in iptables mode, there will be several seconds of timeouts because it flushes the KUBE-SERVICES nat chain and several others. What you expected to happen: Restarting kube-proxy should not impact traffic in any way. How to reproduce it (as minimally and precisely as possible): an HTTP server behind a ClusterIP service an http client in a loop making requests to the HTTP server -15 the kube-proxy process on the client node timeout for a few seconds until kube-proxy re-syncs the iptables rules Anything else we need to know?: Running with makes it pretty clear what's happening: it says it's cleaning up inactive rules, but these are crucial to the iptables proxier. You can trivially reproduce by just running: Environment: Kubernetes version (use ): 1.10.11 Cloud provider or hardware configuration: amazon OS (e.g: ): custom Kernel (e.g. ): 4.14.77-hs623.el6.x86_64 Install tools: custom Others: iptables 1.6.2 We're running kube-proxy 1.10.11, but I've confirmed this issue with kube-proxy 1.13.4 too
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Possibly a bug in the proxy. It flushes iptabes if the proxier is able to use IPVS which seems like the wrong behavior to me if there are shared tables/chains between the IPVS and iptables proxier. One workaround I can think of right now is to unload the ipvs kernel module to skip this check.
/assign
oh, that's an interesting workaround, I'll give that a shot, thanks!
np, I'll try to work on the bug fix in the meantime!
/triage unresolved
note that unloading the ipvs module is not actually enough of a workaround because kube-proxy will load the kernel module if it sees that it is available. If I unload the ipvs module and then rename to something else in then it does the right thing. Dug slightly further: the way it probes for modules is by actually running modprobe.

So the simplest hack that allows us to keep ipvs loaded for other things on the system is to put a modprobe script on its PATH that just always exits 1 for the ip_vs modules. (We can stop mounting the modules dir into the kube-proxy container in our kubernetes clusters, but we also run kube-proxy on non-kubernetes nodes via the init system, so the PATH hack works well enough there). This should hold us over fine until we get to a version with your fix in it.
Good to know, thanks for sharing!
/assign
Quick update on this issue from today's SIG Network call: we're going to try to get rid of the automatic proxy clean up altogether for v1.14.1 since this is considered a bug. is working on the KEP & implementation.


Question: Expect(err).NotTo(HaveOccurred()) }) <del> It("Pods created by ReplicationController should spread to different node", func() { By("Create a pod for each node to make the nodes have almost same cpu/mem usage ratio") createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) By("Create an RC with the same number of replicas as the schedualble nodes. One pod should be scheduled on each node.") config := testutils.RCConfig{ Client:         f.ClientSet, InternalClient: f.InternalClientset, Name:           "scheduler-priority-selector-spreading", Namespace:      ns, Image:          framework.GetPauseImageName(f.ClientSet), Replicas:       len(nodeList.Items), CreatedPods:    &[]*v1.Pod{}, Labels:         map[string]string{"name": "scheduler-priority-selector-spreading"}, CpuRequest:     podRequestedResource.Requests.Cpu().MilliValue(), MemRequest:     podRequestedResource.Requests.Memory().Value(), } Expect(framework.RunRC(config)).NotTo(HaveOccurred()) // Cleanup the replication controller when we are done. defer func() { // Resize the replication controller to zero to get rid of pods. if err := framework.DeleteRCAndPods(f.ClientSet, f.InternalClientset, f.Namespace.Name, "scheduler-priority-selector-spreading"); err != nil { framework.Logf("Failed to cleanup replication controller %v: %v.", "scheduler-priority-selector-spreading", err) } }() pods, err := framework.PodsCreated(f.ClientSet, f.Namespace.Name, "scheduler-priority-selector-spreading", int32(len(nodeList.Items))) Expect(err).NotTo(HaveOccurred()) By("Ensuring each pod is running") result := map[string]bool{} for _, pod := range pods.Items { if pod.DeletionTimestamp != nil { continue } err = f.WaitForPodRunning(pod.Name) Expect(err).NotTo(HaveOccurred()) result[pod.Spec.NodeName] = true framework.PrintAllKubeletPods(cs, pod.Spec.NodeName) } By("Verify the pods were scheduled to each node") if len(nodeList.Items) != len(result) { framework.Failf("Pods are not spread to each node.") } }) </del> It("Pod should be prefer scheduled to node that satisify the NodeAffinity", func() { nodeName := GetNodeThatCanRunPod(f) By("Trying to apply a label on the found node.")

Includes links to these: SchedulerPriorities [Serial] Pods should be scheduled to low resource use rate node SchedulerPriorities [Serial] Pod should be prefer scheduled to node that satisify the NodeAffinity SchedulerPriorities /sig scheduling /assign See also
we need to triage this issue to decide if there are release blocking or not.
This one would be better if you have cycles.
sure, let me try this :).
Among the tests in the first post of this issue, "Pods should be scheduled to low resource use rate node" is failing a lot. As I had mentioned in a different issue, I tried reproducing this locally, but I was not able to. So, my guess is that some configuration/condition/state in our test clusters cause the test to fail. The "least requested" priority function is tested extensively in our unit tests: So, given the issues caused by the test and dependency of scheduling decisions on the config and state of the cluster, I think we can go ahead and delete the e2e test.
Thanks ! Would it help if i broke out the other two failures into separate issues?
Please note that the other two are not failing that frequently. So, I don't see those as blockers for 1.7: https://k8s- That said, as has mentioned in , the other e2e scheduler priority tests should probably be removed eventually. Our e2e test framework is not deterministic enough to run these tests, but given that the other two are passing at a reasonable rate, I would wait until we discuss their removal in our next sig-scheduling meeting.
OK, we're going to ignore 1.5-1.7. Please focus now only on the 1.6 -1.7 tests. PTAL at this . If the view is these test failures aren't "real" failures, but are artifacts of race-conditions in cluster testing, then please delete the tests (since they give us no signal) and devote time to eliminating those race conditions so we can get reliable test coverage.
is also similar to . It is failing a lot in our e2e tests, but have been running the test in my own GCE cluster and has been passing for 18 times so far. I think this one should be deleted as well for similar reasons.


Question: ptrLookup := fmt.Sprintf("%s.in-addr.arpa.", strings.Join(reverseArray(strings.Split(ptrLookupIP, ".")), ".")) ptrRecByUDPFileName := fmt.Sprintf("%s_udp@PTR", ptrLookupIP) ptrRecByTCPFileName := fmt.Sprintf("%s_tcp@PTR", ptrLookupIP) <del> probeCmd += fmt.Sprintf(`test -n "$$(dig +notcp +noall +answer +search %s PTR)" && echo OK > /results/%s;`, ptrLookup, ptrRecByUDPFileName) probeCmd += fmt.Sprintf(`test -n "$$(dig +tcp +noall +answer +search %s PTR)" && echo OK > /results/%s;`, ptrLookup, ptrRecByTCPFileName) </del> <ins> probeCmd += fmt.Sprintf(`check="$$(dig +notcp +noall +answer +search %s PTR)" && test -n "$$check" && echo OK > /results/%s;`, ptrLookup, ptrRecByUDPFileName) probeCmd += fmt.Sprintf(`check="$$(dig +tcp +noall +answer +search %s PTR)" && test -n "$$check" && echo OK > /results/%s;`, ptrLookup, ptrRecByTCPFileName) </ins> fileNames = append(fileNames, ptrRecByUDPFileName) fileNames = append(fileNames, ptrRecByTCPFileName) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: Tests checking DNS responses can have false positives, such as "should provide DNS for services". The tests rely on dig-ing certain domain names [1]: test -n "$(dig +notcp +noall +answer +search kubernetes.default A)" && echo OK /results/ The problems could be that, if a host / container doesn't have any entries in their : ubuntu dig +notcp +noall +answer +search kubernetes.default A ;; connection timed out; no servers could be reached , causing the test to pass and to write "OK" into the results file. [1] What you expected to happen: The tests shouldn't be able to pass if dig failed. How to reproduce it (as minimally and precisely as possible): Comment all nameservers or configure a bogus DNS server in, and run: test -n "$(dig +notcp +noall +answer +search kubernetes.default A)" && echo OK Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.6 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): "Ubuntu 16.04.3 LTS" Kernel (e.g. ): Linux ubuntu 4.4.0-87-generic
/assign
: But one of the purpose of the test is to ensure that the DNS of the cluster is replying. In other words, if it does not reply then we EXPECT this test to fail. If you are not using DNS then isn't it better to SKIP the test ?
I believe means to say "false negatives" ... meaning that in some situations, the test will errantly pass, when it should fail.
oops . Looks like I misread the description, and agree that the dig should work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "strings" "time" <ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/client/clientset_generated/clientset" "k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere" "k8s.io/kubernetes/test/e2e/framework" <del> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" </del> ) func DeleteCinderVolume(name string) error {

https://k8s-
See flakes on . This test needs to be quarantined until it's deflaked.
This test has been moved in the flaky suite ()
Could you please help triage and fix this test? CC
and I are looking into it...
After comparing the relevant passing and failing test logs in the Flaky suite, this appears to just be a timing issue. doesn't give the PD enough time to detach (5 min timeout). Adding above the delete allows a 10 minute detach window before deletion. This is the pattern we've used in other GCEPD tests and have yet to see similar flakes. I'll have a PR up in a bit to add the detach wait. If there's not more flakes for a week after it's merged, I'll submit a PR to pull the Flaky tag.
So I notice this happens when I upgrade a deployment that moves the pod to another node. I can pretty much recreate it at will now. Is the solution to wait 5 minutes before allowing a pod to be created after it was deleted?
After checking the log, there is something wrong at the node side when reconciler tries to unmount the device (global mount path) because there are still some references to the mount. But unmounting local mount point is successfully. Need to investigate more about this issue.


Question: jig.SanityCheckService(svc, v1.ServiceTypeLoadBalancer) framework.ExpectEqual(e2eservice.GetIngressPoint(lbIngress), internalStaticIP) } <del> ginkgo.By("switching to ClusterIP type to destroy loadbalancer") jig.ChangeServiceType(svc.Namespace, svc.Name, v1.ServiceTypeClusterIP, createTimeout) </del> }) // This test creates a load balancer, make sure its health check interval

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Since when has it been failing: It has been failing since 8/28. Here is the k/k diff for additional context: Testgrid link: Reason for failure: Seems like they were two instances of the same test (at least in the latest run). One of them failed with the following error message: The other one with: /milestone v1.16 /priority critical-urgent /kind failing-test /cc /cc /cc
/sig network
- are you aware of any recent changes that might affect this tests (e.g. judging just by title)? It fails deterministically.
/remove-kind bug Actually
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Any chance this is related to ?

This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted ()
can you please let the 1.16 release team if you are going to be able to investigate and provide if you can help an approximate time for a fix?
Acked, I'm looking into the failure and will provide an update soon.
Fantastic! Thanks for the update
Yep, this is related to With service finalizer enabled by default, deleting the namespace (including services) now takes longer --- it now blocks on all the underlying load balancer resources to be cleaned up. It is expected that the latency for resource clean up might be longer on large cluster (such as in this test). I will soon send a PR to overwrite the  with something more reasonable in this case.
Sent to fix this.
Reopening this one to track the job's health /reopen
Reopened this issue. <detailsIn response to : Reopening this one to track the job's health /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yesterday's run has failed due to problems with creating a test cluster. Hopefully we get an answer whether the fix has helped today.
Thanks for tracking this, seems like the latest run is green:
It looks good indeed, thanks /close
Closing this issue. <detailsIn response to : It looks good indeed, thanks </details


Question: func (m *effectiveVersion) Validate() []error { var errs []error // Validate only checks the major and minor versions. <del> binaryVersion := m.binaryVersion.Load().WithPatch(0) </del> <ins> binaryVersion := m.BinaryVersion().WithPatch(0) </ins> emulationVersion := m.emulationVersion.Load() minCompatibilityVersion := m.minCompatibilityVersion.Load()

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: "testing" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields"

cc ref
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- can you please take a look, you introduced this test and it may mean real problem with your code
I took a quick look and this is a test issue. The problem is that the implementation explicitly assumes there is at most one call to dispatchEvent at a given time. And this particular test breaks this assumption: The problem is that, a given iteration is finished when is closed, which may actually happen as a result of even before is called (since it's a different goroutine). And then may be called concurrently with the next iteration, which triggers a race. The fix is simple, we should ensure that all goroutines are finished before starting the next iteration.
/cc


Question: out = klogPrefix.ReplaceAllString(out, "<klog> ") return out } <ins> var initFunc = regexp.MustCompile(`(init.+func|glob.+func)`) // normalizeInitFunctions maps both init.func (used by Go >= 1.22) and // glob..func (used by Go < 1.22) to <init.func>. func normalizeInitFunctions(in string) string { out := initFunc.ReplaceAllString(in, "<init.func>") return out } </ins>

Internal job that runs k8s UT with master golang. https://prow.ppc64le- UT is failing when run with golang tip version. Around 24th of August No response This being a negative test scenario, looking at the test log, it seems the failure message from rest client isn't matching the expected in the string at The part of the text that doesn't match is: Expected: Actual: Steps to reproduce: master golang version clone kubernetes/test/e2e/framework/pod test -v -run TestFailureOutput /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-test /kind cleanup
is it happening in x86 too? is a golang bug ? there is also another bigger problem
Yes it is happening in x86 too. Yes, it could even be a golang bug too. But thought some upgrades in golang might require modification here in k8s. The above panic error that you pointed out already has an issue
This issue is introduced post addition of https://go- Global level functions are getting wrapped by synthetic init and so is getting changed to in actual traceback. Fix strategy has to be decided to support golang change mentioned above. Tagging others to request more inputs. /cc
^^
I think the output test is too strict: it shouldn't depend on the name given to anonymous functions. On the other hand, it should still ensure that functions like are printed as expected. This can be solved by further transforming the text during - I'll prepare a PR. /assign


Question: } func probeNetworkPluginsWithVendorCNIDirPrefix(pluginDir, vendorCNIDirPrefix string) []network.NetworkPlugin { <del> configList := make([]network.NetworkPlugin, 0) network, err := getDefaultCNINetwork(pluginDir, vendorCNIDirPrefix) if err != nil { return configList } return append(configList, &cniNetworkPlugin{ defaultNetwork: network, </del> <ins> plugin := &cniNetworkPlugin{ defaultNetwork: nil, </ins> loNetwork:      getLoNetwork(vendorCNIDirPrefix), execer:         utilexec.New(), <del> }) </del> <ins> } plugin.syncNetworkConfig(pluginDir, vendorCNIDirPrefix) // sync network config from pluginDir periodically to detect network config updates go wait.Forever(func() { plugin.syncNetworkConfig(pluginDir, vendorCNIDirPrefix) }, 10*time.Second) return []network.NetworkPlugin{plugin} </ins> } func ProbeNetworkPlugins(pluginDir string) []network.NetworkPlugin {

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: autoscalingv1 "k8s.io/api/autoscaling/v1" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/apimachinery/pkg/api/errors" apimeta "k8s.io/apimachinery/pkg/api/meta"

What happened: I setup custom metrics based autoscaling with multiple metrics. One of those metrics was not available. HPA reports all metrics as "unknown" (even the ones that are still available) and refuses to operate. What you expected to happen: Pods would be automatically scaled based on metrics that were available, ignoring missing metrics (treating them as zero). A partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has. For example, CPU % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter. If the prometheus adapter goes away we can still use the CPU metric as a lower bound on the number of replicas. If you consider, e.g. , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric. How to reproduce it (as minimally and precisely as possible): Setup a custom metrics server and tell HPA to scale based on several metrics, one of which is not actually available. Environment: kops 1.8 on AWS Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
/sig autoscaling
cc
/kind bug /priority important-longterm
this return returns nil value of statuses variable. statuses variable contains the value of all metrics.
as long as we failed to get one metric of all metrics, the func computeReplicasForMetrics returns, the scale lose efficacy, it is not highly available. when failed to get metric, should we record this event and do not return:)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still having this problem.
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this issue was resolved or it works as designed? thanks,
I was testing this last week and it is not resolved. I don't think it is reasonable to say this is working as designed.
Seems there is a PR to fix it, but it hasn't been reviewed or merged.
yes, I looked into the master branch code, not fixed yet, just curious about it, it would not be fixed or as dobesv said it will be merged/fixed latter with that PR.

/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue was discussed at SIG Autoscaling yesterday: I'm going to raise a new PR later this week given the code's changed so much since the original PR was raised by that it doesn't make sense to just rebase it.
Thanks, that would be excellent!


Question: Subsystem:         KubeletSubsystem, Name:              LastKnownGoodConfigKey, Help:              "The config source the node will fall back to when it encounters certain errors. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question: * nodes (aka 'no') * persistentvolumeclaims (aka 'pvc') * persistentvolumes (aka 'pv') <del> * pods (aka 'po') </del> * poddisruptionbudgets (aka 'pdb') <ins> * pods (aka 'po') </ins> * podsecuritypolicies (aka 'psp') * podtemplates * replicasets (aka 'rs')

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: It is losing in help info. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
will make a pr to address that


Question: ginkgo.By("retrieving the contents of the GMSACredentialSpec custom resource manifest from the node") crdManifestContents := retrieveCRDManifestFileContents(f, node) <del> ginkgo.By("downloading the GMSA webhook deploy script") deployScriptPath, err := downloadFile(gmsaWebhookDeployScriptURL) defer func() { os.Remove(deployScriptPath) }() if err != nil { framework.Failf(err.Error()) } </del> ginkgo.By("deploying the GMSA webhook") <del> webhookCleanUp, err := deployGmsaWebhook(f, deployScriptPath) </del> <ins> webhookCleanUp, err := deployGmsaWebhook(f) </ins> defer webhookCleanUp() if err != nil { framework.Failf(err.Error())

Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing


Question: // GetArticleForNoun returns the article needed for the given noun. func GetArticleForNoun(noun string, padding string) string { <del> if noun[len(noun)-2:] != "ss" && noun[len(noun)-1:] == "s" { </del> <ins> if !strings.HasSuffix(noun, "ss") && strings.HasSuffix(noun, "s") { </ins> // Plurals don't have an article. // Don't catch words like class return fmt.Sprintf("%v", padding)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add a CRD resource with kind only a single character while cause kube-apiserver crashslice bounds out of range What you expected to happen: add validation to limit CRD kind definition or fix the bug of slice out of bounds How to reproduce it (as minimally and precisely as possible): apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: version: v1 names: kind: R plural: rs singular: r scope: Cluster Anything else we need to know?: apiserverlogs: 10:05:32. 6 ] Shutting down OpenAPI controller E6606 10:05:32. 6 ] Observed a panic: runtime.boundsError{x:-1, y:1, signed:true, code:0x3(runtime error: slice bounds out of range [-1:]) goroutine 1670 [running]: (0x41458a0, Gxc00ed66d40) /workspace/anago-v1.17.G- +0xa3 (0x0, 0x0, 0x0) /workspace/anago-v1.17.G- +0x82 panic(0x41458a0, Oxc0Oed66d40) +0x1b2 (0x7493232, 0x1, 0x447d778, 0x1, 0x130, 0xc00ed13680) /workspace/anago-v1.17.0- +0x2b8 ("builder).descriptionFor(€xc007b351f0, 0x0, 0x0, 0x4481982, 0x6, Gxc00ed06d00, 0x18) /workspace/anago-v1.17.0- +0x3c6 ("builder).buildRoute(GxcGG7b351fG, 0xc00ed66d00, 0x18, 0x0, 0x0, 0x447f0F3, 0x4, 0x447f71b, 0x4,

0x4481982, .) /workspace/anago-v1.17.0- .go :281 +0x197 (0xc66b7bcfd0, 0xc005d24a3c, 0x2, 0x101, 0x1, 0x2, Gxc60b76aec8) /workspace/anago-v1.17.0-rc .2.10+70132b0F130acc/src/k8s .io/kubernetes/output/dockerized/go/src/k8s .io/kubernetes/vendor/k8s .io/apiextensions-apiserver/pkg/controller/openapi/builder/builder .go :146 +0x49d (0xc66b7bcfd0, 0x0, Oxb, Gx447f1db, 0x4, 0x1) /workspace/anago-v1.17.0- +0x1a8 ("Controller).Run(0xc0012c8050, Gxc0G1d61200, 0xc007a6d1e0, 0xc069cced80) /workspace/anago-v1.17.0- +0x2da created by /workspace/anago-v1.17.0- .go :236 +0x244 panic: runtime error: slice bounds out of range [-1:] [recovered] panic: runtime error: slice bounds out of range [-1:] Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign
There is a typo in the subject.


Question: discovery "k8s.io/api/discovery/v1" "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/conversion" </ins> "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/sets"

My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21.

My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check.

if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation.

The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: proberesults "k8s.io/kubernetes/pkg/kubelet/prober/results" "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/runtime" <del> "k8s.io/kubernetes/pkg/security/apparmor" </del> kubetypes "k8s.io/kubernetes/pkg/types" "k8s.io/kubernetes/pkg/util/clock" uexec "k8s.io/kubernetes/pkg/util/exec" "k8s.io/kubernetes/pkg/util/flowcontrol" "k8s.io/kubernetes/pkg/util/intstr" "k8s.io/kubernetes/pkg/util/sets" <del> utilstrings "k8s.io/kubernetes/pkg/util/strings" </del> ) type fakeHTTP struct {

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: // This will start reconciliation of node.status.volumesInUse. rc.updateLastSyncTime() } <del> if len(rc.volumesNeedReportedInUse) != 0 && rc.populatorHasAddedPods() { // Once DSW is populated, mark all reconstructed as reported in node.status, // so they can proceed with MountDevice / SetUp. rc.desiredStateOfWorld.MarkVolumesReportedInUse(rc.volumesNeedReportedInUse) rc.volumesNeedReportedInUse = nil } </del> }

remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign


Question: RuntimeSpec: RuntimeSpec{ DockerSpec: &DockerSpec{ Version:     []string{`1.1[1-3]..*`, `17.0[3,6,9]..*`, `18.0[6,9]..*`, `19.03..*`, `20.10..*`}, <del> GraphDriver: []string{"aufs", "overlay", "overlay2", "devicemapper", "zfs"}, </del> <ins> GraphDriver: []string{"aufs", "btrfs", "overlay", "overlay2", "devicemapper", "zfs"}, </ins> }, }, }

every k8s release, if changes are to we release a new version of the library and vendor it in this repository. TODO: [ ] wait for changes to accumulate in [ ] add new release in before the k8s 1.23 release [ ] vendor the new release in k/k (example PR: )
/sig cluster-lifecycle node /milestone v1.23 /assign
/triage accepted
/cc


Question: } // newPodInformer creates a shared index informer that returns only non-terminal pods. <ins> // The PodInformer allows indexers to be added, but note that only non-conflict indexers are allowed. </ins> func newPodInformer(cs clientset.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer { selector := fmt.Sprintf("status.phase!=%v,status.phase!=%v", v1.PodSucceeded, v1.PodFailed) tweakListOptions := func(options *metav1.ListOptions) { options.FieldSelector = selector } <del> return coreinformers.NewFilteredPodInformer(cs, metav1.NamespaceAll, resyncPeriod, nil, tweakListOptions) </del> <ins> return coreinformers.NewFilteredPodInformer(cs, metav1.NamespaceAll, resyncPeriod, cache.Indexers{}, tweakListOptions) </ins> }

I'm writing a scheduler plugin using the scheduler framework, and in my plugin I'd like to add a using the provided by the . Adding such a Indexer lead to a runtime crash like The reason for this is that I'm not sure the scheduler framework wants to explicitely disallow adding Indexers using the facilities. If this is the case, it should be more explicit somehow because the current behaviour is surprising. If the scheduler framework doesn't want to add this limitation, then adding indexers should not lead to runtime crashes add a new scheduler plugin trying to add indexers. Sample code (relevant minimal portions only) I'm not sure this is a real bug and not a deliberate decision; a research was inconcusive, and a (admittedly quick) search in the past issues was not fruitful either. The fix could be as trivial as <detailsconsole $ kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-10T16:30:48Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"} </detailsbut applies also on latest master (see above) <detailsany </details<detailsany </details<detailsnot relevant </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: } dryRun := cmdutil.GetFlagBool(cmd, "dry-run") <ins> output := cmdutil.GetFlagString(cmd, "output") </ins> count := 0 err = r.Visit(func(info *resource.Info, err error) error {

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: } }) <del> // Regression test for #74839, where: // Packets considered INVALID by conntrack are now dropped. In particular, this fixes // a problem where spurious retransmits in a long-running TCP connection to a service // IP could result in the connection being closed with the error "Connection reset by // peer" ginkgo.It("should resolve connection reset issue #74839 [Slow]", func() { serverLabel := map[string]string{ "app": "boom-server", } clientLabel := map[string]string{ "app": "client", } serverPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "boom-server", Labels: serverLabel, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "boom-server", Image: imageutils.GetE2EImage(imageutils.RegressionIssue74839), Ports: []v1.ContainerPort{ { ContainerPort: 9000, // Default port exposed by boom-server }, }, }, }, Affinity: &v1.Affinity{ PodAntiAffinity: &v1.PodAntiAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ { LabelSelector: &metav1.LabelSelector{ MatchLabels: clientLabel, }, TopologyKey: "kubernetes.io/hostname", }, }, }, }, }, } _, err := fr.ClientSet.CoreV1().Pods(fr.Namespace.Name).Create(context.TODO(), serverPod, metav1.CreateOptions{}) framework.ExpectNoError(err) err = e2epod.WaitForPodsRunningReady(fr.ClientSet, fr.Namespace.Name, 1, 0, framework.PodReadyBeforeTimeout, map[string]string{}) framework.ExpectNoError(err) ginkgo.By("Server pod created") svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "boom-server", }, Spec: v1.ServiceSpec{ Selector: serverLabel, Ports: []v1.ServicePort{ { Protocol: v1.ProtocolTCP, Port:     9000, }, }, }, } _, err = fr.ClientSet.CoreV1().Services(fr.Namespace.Name).Create(context.TODO(), svc, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Server service created") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "startup-script", Labels: clientLabel, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "startup-script", Image: imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{ "sh", "-c", "while true; do sleep 2; nc boom-server 9000& done", }, }, }, Affinity: &v1.Affinity{ PodAntiAffinity: &v1.PodAntiAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ { LabelSelector: &metav1.LabelSelector{ MatchLabels: serverLabel, }, TopologyKey: "kubernetes.io/hostname", }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } _, err = fr.ClientSet.CoreV1().Pods(fr.Namespace.Name).Create(context.TODO(), pod, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Client pod created") for i := 0; i < 20; i++ { time.Sleep(3 * time.Second) resultPod, err := fr.ClientSet.CoreV1().Pods(fr.Namespace.Name).Get(context.TODO(), serverPod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) gomega.Expect(resultPod.Status.ContainerStatuses[0].LastTerminationState.Terminated).Should(gomega.BeNil()) } }) </del> })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func newServiceInfo(serviceName proxy.ServicePortName, port *api.ServicePort, service *api.Service) *serviceInfo { onlyNodeLocalEndpoints := false if utilfeature.DefaultFeatureGate.Enabled(features.ExternalTrafficLocalOnly) && <del> (service.Spec.Type == api.ServiceTypeLoadBalancer || service.Spec.Type == api.ServiceTypeNodePort) && apiservice.NeedsHealthCheck(service) { </del> <ins> apiservice.RequestsOnlyLocalTraffic(service) { </ins> onlyNodeLocalEndpoints = true } info := &serviceInfo{

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: "k8s.io/klog" <ins> corev1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/version" e2elog "k8s.io/kubernetes/test/e2e/framework/log"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which test(s) are failing: When is specified in kubetest, the is trying to delete all namespaces except , and (see ) With Kubernetes 1.14 the features was enabled which creates their lease objects in a separate namespace which could sometimes not be cleaned up correctly. () Since when has it been failing: Testgrid link: Reason for failure: The SetupSuite is trying to delete system namespace Anything else we need to know:
/sig testing


Question: t.Errorf("Expected error, got none") } if !strings.Contains(err.Error(), <del> "unattached volumes=[vol2], failed to process volumes=[vol3]") { </del> <ins> "unattached volumes=[vol02 vol2], failed to process volumes=[vol03 vol3]") { </ins> t.Errorf("Unexpected error info: %v", err) } }

Currently I describe a Pod that is in and fails to attach/mounts its volume: Note that the generated events are shown in 4 section. The only difference between the messages is how the are ordered. I suggest to sort the unmounted/unattached) volumes for the event message. This will allow to group all events and properly have the age and the occurrences count reported. I would expect the unmounted/unattached) volumes to be sorted in the events' message and to do not have several group of events for the same issue. See above. No response <details</details<detailsAWS but does not matter for this issue. </details<details</details<details</details<details</details<details</details
/sig storage
/assign qingwave
/triage accepted


Question: }, ) // StartedPodsErrorsTotal is a counter that tracks the number of errors creating pod sandboxes <del> StartedPodsErrorsTotal = metrics.NewCounterVec( </del> <ins> StartedPodsErrorsTotal = metrics.NewCounter( </ins> &metrics.CounterOpts{ Subsystem:      KubeletSubsystem, Name:           StartedPodsErrorsTotalKey, Help:           "Cumulative number of errors when starting pods", StabilityLevel: metrics.ALPHA, }, <del> []string{"message"}, </del> ) // StartedContainersTotal is a counter that tracks the number of container creation operations StartedContainersTotal = metrics.NewCounterVec(

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --While reviewing labels in prometheus noticed this metric whose label value is large (700+ characters) since it was a error log message. Example: The metric is from It doesn't look like we should have error/ log message as a metric Query the metric in prometheus UI or through api where kubelet is one of the target in prometheus Kubernetes version (use ): v1.22.0-rc.0 Cloud provider or hardware configuration: NA OS (e.g: ): NA Kernel (e.g. ): NA Install tools: Network plugin and version (if this is a network-related bug): Others: cc:
I agree that error logs should never be part of metrics, it is not the correct vector to convey this kind of information. Logging would be more appropriated here. /sig instrumentation /sig node /cc
I want to work on this. /assign
/triage accepted
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // getGPUDevicePluginImage returns the image of GPU device plugin. <del> func getGPUDevicePluginImage() string { </del> <ins> func getGPUDevicePluginImage() (string, error) { </ins> ds, err := e2emanifest.DaemonSetFromURL(e2egpu.GPUDevicePluginDSYAML) if err != nil { <del> klog.Errorf("Failed to parse the device plugin image: %v", err) return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: %w", err) </ins> } if ds == nil { <del> klog.Errorf("Failed to parse the device plugin image: the extracted DaemonSet is nil") return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: the extracted DaemonSet is nil") </ins> } if len(ds.Spec.Template.Spec.Containers) < 1 { <del> klog.Errorf("Failed to parse the device plugin image: cannot extract the container from YAML") return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: cannot extract the container from YAML") </ins> } <del> return ds.Spec.Template.Spec.Containers[0].Image </del> <ins> return ds.Spec.Template.Spec.Containers[0].Image, nil </ins> } // getSRIOVDevicePluginImage returns the image of SRIOV device plugin. <del> func getSRIOVDevicePluginImage() string { </del> <ins> func getSRIOVDevicePluginImage() (string, error) { </ins> data, err := e2etestfiles.Read(SRIOVDevicePluginDSYAML) if err != nil { <del> klog.Errorf("Failed to read the device plugin manifest: %v", err) return "" </del> <ins> return "", fmt.Errorf("failed to read the device plugin manifest: %w", err) </ins> } ds, err := e2emanifest.DaemonSetFromData(data) if err != nil { <del> klog.Errorf("Failed to parse the device plugin image: %v", err) return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: %w", err) </ins> } if ds == nil { <del> klog.Errorf("Failed to parse the device plugin image: the extracted DaemonSet is nil") return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: the extracted DaemonSet is nil") </ins> } if len(ds.Spec.Template.Spec.Containers) < 1 { <del> klog.Errorf("Failed to parse the device plugin image: cannot extract the container from YAML") return "" </del> <ins> return "", fmt.Errorf("failed to parse the device plugin image: cannot extract the container from YAML") </ins> } <del> return ds.Spec.Template.Spec.Containers[0].Image </del> <ins> return ds.Spec.Template.Spec.Containers[0].Image, nil </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig node /triage accepted


Question: "time" "k8s.io/apiserver/pkg/server/dynamiccertificates" <ins> "k8s.io/component-base/cli/flag" </ins> "k8s.io/kubernetes/cmd/kube-apiserver/app/options" "k8s.io/kubernetes/test/integration/framework" )

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details