Question: type process struct{ name, file string } dockerProcs := []process{{dockerProcessName, dockerPidFile}} if dockerAPIVersion.AtLeast(containerdAPIVersion) { <ins> // By default containerd is started separately, so there is no pid file. containerdPidFile := "" </ins> dockerProcs = append(dockerProcs, process{containerdProcessName, containerdPidFile}) } var errs []error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: docker-containerd have renamed to containerd, but EnsureDockerInContainer check not update,so it not work What you expected to happen: Is this check still needed How to reproduce it (as minimally and precisely as possible): no need Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
cc
feel free to send an PR update with new name for the process and the location of the pid file.


Question: }, "ownerReferences": []map[string]interface{}{ { <del> "kind":               "Pod", "name":               "poda", "apiVersion":         "v1", "uid":                "1", "controller":         (*bool)(nil), "blockOwnerDeletion": (*bool)(nil), </del> <ins> "kind":       "Pod", "name":       "poda", "apiVersion": "v1", "uid":        "1", </ins> }, { "kind":               "Pod", "name":               "podb", "apiVersion":         "v1", "uid":                "2", <del> "controller":         &trueVar, "blockOwnerDeletion": &trueVar, </del> <ins> "controller":         true, "blockOwnerDeletion": true, </ins> }, }, "finalizers": []interface{}{

Kubernetes version Code from master. What happened: does not return and fields after they are set to non-nil values using . How to reproduce it (as minimally and precisely as possible): See PR
/sig api-machinery
This is a good find and I don't think I saw the notification come through :(


Question: DoRaw() } <del> for i := 0; !passed; i++ { // Timeout will keep us from going forever. </del> <ins> timeout := time.Now().Add(2 * time.Minute) for i := 0; !passed && timeout.After(time.Now()); i++ { </ins> time.Sleep(2 * time.Second) Logf("About to make a proxy status call") start := time.Now()

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package capabilities import ( "reflect" "testing" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/extensions" ) func TestGenerateAdds(t *testing.T) { tests := map[string]struct { defaultAddCaps []api.Capability containerCaps  *api.Capabilities expectedCaps   *api.Capabilities }{ "no required, no container requests": {}, "no required, no container requests, non-nil": { containerCaps: &api.Capabilities{}, expectedCaps:  &api.Capabilities{}, }, "required, no container requests": { defaultAddCaps: []api.Capability{"foo"}, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, container requests add required": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "multiple required, container requests add required": { defaultAddCaps: []api.Capability{"foo", "bar", "baz"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "baz", "foo"}, }, }, "required, container requests add non-required": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "foo"}, }, }, "generation does not mutate unnecessarily": { defaultAddCaps: []api.Capability{"foo", "bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo", "foo", "bar", "baz"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo", "foo", "bar", "baz"}, }, }, "generation dedupes": { defaultAddCaps: []api.Capability{"foo", "bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo", "baz"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "baz", "foo"}, }, }, "generation is case sensitive - will not dedupe": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"FOO"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"FOO", "foo"}, }, }, } for k, v := range tests { container := &api.Container{ SecurityContext: &api.SecurityContext{ Capabilities: v.containerCaps, }, } strategy, err := NewDefaultCapabilities(v.defaultAddCaps, nil, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } generatedCaps, err := strategy.Generate(nil, container) if err != nil { t.Errorf("%s failed generating: %v", k, err) continue } if v.expectedCaps == nil && generatedCaps != nil { t.Errorf("%s expected nil caps to be generated but got %v", k, generatedCaps) continue } if !reflect.DeepEqual(v.expectedCaps, generatedCaps) { t.Errorf("%s did not generate correctly.  Expected: %#v, Actual: %#v", k, v.expectedCaps, generatedCaps) } } } func TestGenerateDrops(t *testing.T) { tests := map[string]struct { defaultAddCaps   []api.Capability requiredDropCaps []api.Capability containerCaps    *api.Capabilities expectedCaps     *api.Capabilities }{ "no required, no container requests": { expectedCaps: nil, }, "no required, no container requests, non-nil": { containerCaps: &api.Capabilities{}, expectedCaps:  &api.Capabilities{}, }, "required drops are defaulted": { requiredDropCaps: []api.Capability{"foo"}, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "required drops are defaulted when making container requests": { requiredDropCaps: []api.Capability{"baz"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "baz", "foo"}, }, }, "required drops do not mutate unnecessarily": { requiredDropCaps: []api.Capability{"baz"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar", "baz"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar", "baz"}, }, }, "can drop a required add": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "can drop non-required add": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar"}, }, expectedCaps: &api.Capabilities{ Add:  []api.Capability{"foo"}, Drop: []api.Capability{"bar"}, }, }, "defaulting adds and drops, dropping a required add": { defaultAddCaps:   []api.Capability{"foo", "bar", "baz"}, requiredDropCaps: []api.Capability{"abc"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add:  []api.Capability{"bar", "baz"}, Drop: []api.Capability{"abc", "foo"}, }, }, "generation dedupes": { requiredDropCaps: []api.Capability{"baz", "foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "foo"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "baz", "foo"}, }, }, "generation is case sensitive - will not dedupe": { requiredDropCaps: []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"BAR"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"BAR", "bar"}, }, }, } for k, v := range tests { container := &api.Container{ SecurityContext: &api.SecurityContext{ Capabilities: v.containerCaps, }, } strategy, err := NewDefaultCapabilities(v.defaultAddCaps, v.requiredDropCaps, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } generatedCaps, err := strategy.Generate(nil, container) if err != nil { t.Errorf("%s failed generating: %v", k, err) continue } if v.expectedCaps == nil && generatedCaps != nil { t.Errorf("%s expected nil caps to be generated but got %#v", k, generatedCaps) continue } if !reflect.DeepEqual(v.expectedCaps, generatedCaps) { t.Errorf("%s did not generate correctly.  Expected: %#v, Actual: %#v", k, v.expectedCaps, generatedCaps) } } } func TestValidateAdds(t *testing.T) { tests := map[string]struct { defaultAddCaps []api.Capability allowedCaps    []api.Capability containerCaps  *api.Capabilities expectedError  string }{ // no container requests "no required, no allowed, no container requests": {}, "no required, allowed, no container requests": { allowedCaps: []api.Capability{"foo"}, }, "required, no allowed, no container requests": { defaultAddCaps: []api.Capability{"foo"}, expectedError:  `capabilities: Invalid value: "null": required capabilities are not set on the securityContext`, }, // container requests match required "required, no allowed, container requests valid": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, no allowed, container requests invalid": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedError: `capabilities.add: Invalid value: "bar": capability may not be added`, }, // container requests match allowed "no required, allowed, container requests valid": { allowedCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "no required, all allowed, container requests valid": { allowedCaps: []api.Capability{extensions.AllowAllCapabilities}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "no required, allowed, container requests invalid": { allowedCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedError: `capabilities.add: Invalid value: "bar": capability may not be added`, }, // required and allowed "required, allowed, container requests valid required": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, allowed, container requests valid allowed": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, }, "required, allowed, container requests invalid": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"baz"}, }, expectedError: `capabilities.add: Invalid value: "baz": capability may not be added`, }, "validation is case sensitive": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"FOO"}, }, expectedError: `capabilities.add: Invalid value: "FOO": capability may not be added`, }, } for k, v := range tests { strategy, err := NewDefaultCapabilities(v.defaultAddCaps, nil, v.allowedCaps) if err != nil { t.Errorf("%s failed: %v", k, err) continue } errs := strategy.Validate(nil, nil, v.containerCaps) if v.expectedError == "" && len(errs) > 0 { t.Errorf("%s should have passed but had errors %v", k, errs) continue } if v.expectedError != "" && len(errs) == 0 { t.Errorf("%s should have failed but received no errors", k) continue } if len(errs) == 1 && errs[0].Error() != v.expectedError { t.Errorf("%s should have failed with %v but received %v", k, v.expectedError, errs[0]) continue } if len(errs) > 1 { t.Errorf("%s should have failed with at most one error, but received %v: %v", k, len(errs), errs) } } } func TestValidateDrops(t *testing.T) { tests := map[string]struct { requiredDropCaps []api.Capability containerCaps    *api.Capabilities expectedError    string }{ // no container requests "no required, no container requests": {}, "required, no container requests": { requiredDropCaps: []api.Capability{"foo"}, expectedError:    `capabilities: Invalid value: "null": required capabilities are not set on the securityContext`, }, // container requests match required "required, container requests valid": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "required, container requests invalid": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar"}, }, expectedError: `capabilities.drop: Invalid value: []core.Capability{"bar"}: foo is required to be dropped but was not found`, }, "validation is case sensitive": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"FOO"}, }, expectedError: `capabilities.drop: Invalid value: []core.Capability{"FOO"}: foo is required to be dropped but was not found`, }, } for k, v := range tests { strategy, err := NewDefaultCapabilities(nil, v.requiredDropCaps, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } errs := strategy.Validate(nil, nil, v.containerCaps) if v.expectedError == "" && len(errs) > 0 { t.Errorf("%s should have passed but had errors %v", k, errs) continue } if v.expectedError != "" && len(errs) == 0 { t.Errorf("%s should have failed but received no errors", k) continue } if len(errs) == 1 && errs[0].Error() != v.expectedError { t.Errorf("%s should have failed with %v but received %v", k, v.expectedError, errs[0]) continue } if len(errs) > 1 { t.Errorf("%s should have failed with at most one error, but received %v: %v", k, len(errs), errs) } } } </ins>

The name of this file is odd: The file context has nothing to do with Is this a BUG REPORT or FEATURE REQUEST?: /assign


Question: Priority                          *int32 PreemptionPolicy                  *v1.PreemptionPolicy PriorityClassName                 string <ins> Volumes                           []v1.Volume </ins> } // InitPausePod initializes a pod API object from the given config. It is used

We need an integration test for the scheduler enforcement of the alpha feature. This is required for beta graduation; we need to collect signal on the current behavior before we can proceed. See this comment for context:
/sig storage /sig scheduler
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig storage /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: "cgroup_manager_test.go", "container_manager_linux_test.go", "helpers_linux_test.go", <del> "node_container_manager_test.go", </del> <ins> "node_container_manager_linux_test.go", </ins> "pod_container_manager_linux_test.go", ], embed = [":go_default_library"],

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: "strings" "testing" <ins> "github.com/spf13/cobra" </ins> "k8s.io/cli-runtime/pkg/genericclioptions" cmdutil "k8s.io/kubectl/pkg/cmd/util" )

What happened: I am using a command like below in GitHub actions: I get the following error: What you expected to happen: Since I am using flag, it should work without requiring a valid kubeconfig file. How to reproduce it (as minimally and precisely as possible): Just delete your local kubeconfig file and try the above command. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign /kind regression
This likely breaks some scripted usage of kubectl
This is related to /assign
Or
Yes, sorry I mentioned you in this issue. Because my first thought is this brings and which will check the config file existence but in fact that's not the root cause. Actually the root cause This checks config has the server or not. And with this , this function will return error when there's no local config file. I think there might be two solutions? the similar logic like to every , , ..... when or is set. Seems not a good idea. reverse this to have the default server which is better than ? By the way, is equals ? Since This works only for .
The plan I agreed on with Jordan is that I will revert since that seems to be the main culprit behind all these problems. I'll add a test ensuring this works as required. This will allow us to backport the fix to 1.18. For 1.19 I'll put together a proper fix supporting and flags.


Question: defer os.RemoveAll(socketDir) m, p := setup(t, []*pluginapi.Device{}, func(n string, a, u, r []pluginapi.Device) {}, socketName, pluginSocketName) cleanup(t, m, p) <ins> // Stop should tolerate being called more than once. cleanup(t, m, p) } func TestNewManagerImplStop(t *testing.T) { socketDir, socketName, pluginSocketName, err := tmpSocketDir() require.NoError(t, err) defer os.RemoveAll(socketDir) m, err := newManagerImpl(socketName) require.NoError(t, err) // No prior Start, but that should be okay. err = m.Stop() require.NoError(t, err) devs := []*pluginapi.Device{ {ID: "Dev1", Health: pluginapi.Healthy}, {ID: "Dev2", Health: pluginapi.Healthy}, } p := NewDevicePluginStub(devs, pluginSocketName) // Same here. err = p.Stop() require.NoError(t, err) </ins> } // Tests that the device plugin manager correctly handles registration and re-registration by

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: Read more about CRI API at [kubernetes docs](https://kubernetes.io/docs/concepts/architecture/cri/). The repository [kubernetes/cri-api](https://github.com/kubernetes/cri-api) is a mirror of https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cri-api. <del> Please do NOT file issues or submit PRs against the [kubernetes/cri-api](https://github.com/kubernetes/cri-api) </del> <ins> Please do **not** file issues or submit PRs against the [kubernetes/cri-api](https://github.com/kubernetes/cri-api) </ins> repository as it is readonly, all development is done in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes). <ins> The CRI API is defined in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) repository and is **only** intended to be used for kubelet to container runtime interactions, or for node-level troubleshooting using a tool such as `crictl`. It is **not** a common purpose container runtime API for general use, and is intended to be Kubernetes-centric. We try to avoid it, but there may be logic within a container runtime that optimizes for the order or specific parameters of call(s) that the kubelet makes. ## Version skew policy On a single Node there may be installed multiple components implementing different versions of CRI API. For example, on a single node there might be: - _Kubelet_ may call into _Container Runtime_ (e.g. [containerd](https://containerd.io)) and _Image Service Proxy_ (e.g. [stargz-snapshotter](https://github.com/containerd/stargz-snapshotter)). _Container Runtime_ may be versioned with the OS Image, _Kubelet_ is installed by system administrator and _Image Service proxy_ is versioned by the third party vendor. - _Image Service Proxy_ calls into _Container Runtime_. - _CRI tools_ (e.g. [crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/)) may be installed by end user to troubleshoot, same as a third party daemonsets. All of them are used to call into the _Container Runtime_ to collect container information. So on a single node it may happen that _Container Runtime_ is serving a newer version'd kubelet and older versioned crictl. This is a supported scenario within the version skew policy. ### Version Skew Policy for CRI API CRI API has two versions: - Major semantic version (known versions are `v1alpha2` ([removed in 1.26](https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed)), `v1`). - Kubernetes version (for example: `@1.23`). Note, the `cri-api` Golang library is versioned as `0.23` as it doesn't guarantee Go types backward compatibility. Major semantic version (e.g. `v1`) is used to introduce breaking changes and major new features that are incompatible with the current API. Kubernetes version is used to indicate a specific feature set implemented on top of the major semantic version. All changes made without the change of a major semantic version API must be backward and forward compatible. - _Kubelet_ must work with the older _Container Runtime_ if it implements the same semantic version of CRI API (e.g. `v1`) of up to three Kubernetes minor versions back. New features implemented in CRI API must be gracefully degraded. For example, _Kubelet_ of version 1.26 must work with _Container Runtime_ implementing `k8s.io/cri-api@v0.23.0`+. - _Kubelet_ must work with _Container Runtime_ if it implements the same semantic version of CRI API (e.g. `v1`) of up to three minor versions up. New features implemented in CRI API must not change behavior of old method calls and response values. For example, _Kubelet_ of version 1.22 must work with _Container Runtime_ implementing `k8s.io/cri-api@v0.25.5`. ## Versioning This library contains go classes generated from the CRI API protocol buffers and gRPC API. The library versioned as `0.XX` as Kubernetes doesn't provide any guarantees on backward compatibility of Go wrappers between versions. However CRI API itself (protocol buffers and gRPC API) is marked as stable `v1` version and it is backward compatible between versions. Versions like `v0.<minor>.<patch>` (e.g. `v0.25.5`) are considered stable. It is discouraged to introduce CRI API changes in patch releases and recommended to use versions like `v0.<minor>.0`. All alpha and beta versions (e.g. `k8s.io/cri-api@v0.26.0-beta.0`) should be backward and forward compatible. ## Feature development Some features development requires changes in CRI API and corresponding changes in _Container Runtime_. Coordinating between Kubernetes branches and release versions and _Container Runtime_ versions is not always trivial. The recommended feature development flow is following: - Review proposed CRI API changes during the KEP review stage. Some field names and types may not be spelled out exactly at this stage. - Locally implement a prototype that implement changes in both - Kubernetes and Container Runtime. - Submit a Pull Request for Kubernetes implementing CRI API changes alongside the feature code. Feature must be developed to degrade gracefully when used with older Container Runtime according to the Version Skew policy. - Once PR is merged, wait for the next Kubernetes release tag being produced. Find the corresponding CRI API tag (e.g. `k8s.io/cri-api@v0.26.0-beta.0`). - This tag can be used to implement the feature in Container Runtime. It is recommended to switch to the stable tag like (`k8s.io/cri-api@v0.26.0`) once available. ## Change history Here is the change history of the Container Runtime Interface protocol: ### v1.20 `git diff v1.19.0 v1.20.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - CRI [v1 introduced](https://github.com/kubernetes/kubernetes/pull/96387) ### v1.21 `git diff v1.20.0 v1.21.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` No changes ### v1.22 `git diff v1.21.0 v1.22.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [Windows host process support](https://github.com/kubernetes/kubernetes/pull/99576) - `PodSandboxConfig` has `windows` field  of type `WindowsPodSandboxConfig` - New type `WindowsPodSandboxConfig` introduced - New type `WindowsSandboxSecurityContext` introduced - The type `WindowsContainerSecurityContext` has a new `host_process` boolean field - [Feature: add unified on CRI to support cgroup v2](https://github.com/kubernetes/kubernetes/pull/102578) - The type `LinuxContainerResources` has a new field `unified` which is a map of strings - [Alpha node swap support](https://github.com/kubernetes/kubernetes/pull/102823) - The type `LinuxContainerResources` has a new `memory_swap_limit_in_bytes` int64 field ### v1.23 `git diff v1.22.0 v1.23.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [CRI: add fields for pod level stats to satisfy the /stats/summary API](https://github.com/kubernetes/kubernetes/pull/102789) - New functions `PodSandboxStats`, `ListPodSandboxStats` with the corresponding types of request and response objects are introduced - [pass sandbox resource requirements over CRI](https://github.com/kubernetes/kubernetes/pull/104886) - New fields on `LinuxPodSandboxConfig`: `overhead` and `resources` of type `LinuxContainerResources`. - [prevents garbage collection from removing pinned images](https://github.com/kubernetes/kubernetes/pull/103299) - The type `Image` has a new boolean field `pinned` ### v1.24 `git diff v1.23.0 v1.24.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [Update CRI-API Capabilities to include a field that allows us to set ambient capabilities](https://github.com/kubernetes/kubernetes/pull/104620) - The type `Capability` has a new string field `add_ambient_capabilities` - [CRI-API - Add rootfs size to WindowsContainerResources](https://github.com/kubernetes/kubernetes/pull/108894) - The type `WindowsContainerResources` has a new int64 field `rootfs_size_in_bytes` ### v1.25 `git diff v1.24.0 v1.25.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [kubelet: add CRI definitions for user namespaces](https://github.com/kubernetes/kubernetes/pull/110535) - The new type `UserNamespace` introduced to represent user namespaces id mapping - The type `NamespaceOption` has a new field `userns_options` of type `UserNamespace` - [Minimal checkpointing support](https://github.com/kubernetes/kubernetes/pull/104907) - The new method `CheckpointContainer` introduced with the corresponding request and response types - [Update CRI API to support Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111642) - The new streaming method `GetContainerEvents` is introduced with the corresponding request and response types - [CRI changes to support in-place pod resize](https://github.com/kubernetes/kubernetes/pull/111645) - The new type `ContainerResources` is introduced - The type `ContainerStatus` has a new field `resources` of type `ContainerResources` - The semantic of `UpdateContainerResources` updated. The method must be implemented as synchronous and return error on failure ### v1.26 `git diff v1.25.0 v1.26.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [CRI: Add Windows Podsandbox Stats](https://github.com/kubernetes/kubernetes/pull/110754) - Added fields to the type `WindowsPodSandboxStats` expressing stats required to be collected from windows pods. - [Windows hostnetwork alpha](https://github.com/kubernetes/kubernetes/pull/112961) - New type `WindowsNamespaceOption` introduced - The type `WindowsSandboxSecurityContext` has a new field `namespace_options` of type `WindowsNamespaceOption` - [Improve the API description of `PodSecurityContext.SupplementalGroups` to clarify its unfamiliar behavior](https://github.com/kubernetes/kubernetes/pull/113047) - Clarified the expected behavior of `SupplementalGroups` field of `PodSecurityContext` - [Add Support for Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111384) - The type `ContainerEventResponse` updated: the field `pod_sandbox_metadata` removed and fields `pod_sandbox_status` and `containers_statuses` added. - The type `PodSandboxStatusResponse` has a new fields `containers_statuses` and `timestamp` </ins> ## Community, discussion, contribution, and support Learn how to engage with the Kubernetes community on the [community

We need to define the CRI API versioning policies: we bump the version of API when methods and optional fields are ? CRI API need to indicate that the new methods are optional (not part of a conformance) for a new features that are in alpha/beta? the release cycle for the features going thru alpha/beta stages that require CRI API changes and implementation in various runtimes. CRI API allow "optional features like suggested here: Would this needs to be tested as part of k8s e2e to make sure k8s doesn't depend on "optional" APIs. Also we need to define the deprecation and feature removal policies. We need to define clear rules of working with CRI API. /sig node /cc /priority important-longterm
+1 Thanks for raising this issue.
Great point, I'm in favor of this. With v1 we're in shape to define the right set of policies. To clarify, when speaking about the version, we're referring to the runtime version defined there: No, because in protobuf 3 all fields are optional and we should only bump the CRI version on breaking changes. Adding new services (methods) which are not supported by the runtime would return an "Unimplemented" error, which can be handed by the kubelet. Yeah, by adding documentation to k/website?
IMO no. While I understand the "we linked them together and it didn't work" problem, that feels like a problem of their own making. The deps constraints (implicit or explicit, I don't know) are being violated. Now, we COULD CHOOSE to jump through hoops to make this better - is it worth the energy to do this? Once we do it, it will be hard to stop doing it. E.g. defining a new for every method we add. Hopefully we don't add a lot of methods... I don't understand what you mean here? Yes. This is the biggest issue with out-of-tree implementations - adding a feature to the API is easy, but getting impls to support it, then cut a release, etc is harder. We lack a clear set of rules - do they HAVE TO support a feature? Do they have to support it pre-GA? Will we block GA contingent on impl support? How do we get feedback on new features?

This problem exists in networking and storage, too. Unless we plan to require every impl to support every feature, we need some way to present and/or query what works and what doesn't, right? Otherwise we end up with impls that are lacking feature X but still accept the API for it - a recipe for unhappy users.
Adding in a reference to https://gateway-
For storage we make most features optional. If in the future we require some new feature (this hasn't happened yet), I think we would have to bump the major version. User discoverability of what optional features an implementation supports remains a challenge. Today each implementation is responsible for documenting which features they support. We have a to list implementations, but it's up to each implementation to keep their entry updated. Right now if a user tries to use an optional feature that the implementation doesn't support, they will get feedback ranging from a nice error message to no progress and no response.
I think it might be interesting to have at least a common pattern for this.
I have started this document for the discussion:
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen </details


Question: localhostNodePorts:       localhostNodePorts, nodePortAddresses:        nodePortAddresses, networkInterfacer:        proxyutil.RealNetwork{}, <ins> conntrackTCPLiberal:      conntrackTCPLiberal, </ins> } burstSyncs := 2

When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. However, there is no corresponding session link on the host, and the host sends a reset packet, causing the session link to be interrupted connection not reset This problem can be solved by command iptables -t filter -I INPUT -p tcp -m conntrack --ctstate INVALID -j DROP Similar But this issue "drop" is placed on the forward chain, our scenario needs to be placed on the input chain. $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} <details</details<details</details<details</details<details</details<details</details
/sig network is this about pods with hostNetwork: true ? if not, why is this a kube-proxy issue if the problem is in INPUT?
/assign
Yeah, to be clear - filter INPUT applies to packets which are going to be delivered to a process. Is that kube-proxy's job to define?
The "How can we reproduce" sounds a bit like this always happens. Here is a better instruction
IMO is the best description of the problem:
This may be a better solution than let add rules in the INPUT chain
hehe, ironically we should decide to or , one is a sysctl call the other is one (or two) iptables rules. The iptables rule has also the side effect of breaking people (cc ) If we switch to set the option, the change does not seem to be backportable. I tend to agree with Lars, remove the iptables INVALID drop rule and set the sysctl, we are already setting more host sysctls what do you think?

it will be nice to have some iptables hackers opinion too
I tried to recreate this problem on virtual en (kvm/qemu) by requsting a 100MB file with curl and do; to provoke some packet loss. The transfer took longer time, but never failed.
IMO kube-proxy should not set any sysctls that are not literally required for functionality that the user has explicitly opted into (eg for most network plugins). Kube-proxy does not own the host network namespace, and it should not be doing things that will affect other people's host-network traffic, because if we do it's going to break some users. (See also: .) We can document that we think it's a good idea for users/distros to set , and if we think it's almost always a better idea than not, we can warn at startup if it's not set, but (IMO) we shouldn't set it ourselves. If we can get the same effect with an iptables rule that only affects kube-proxy's own traffic, then I think that's better than setting a sysctl that will also affect non-kube-proxy traffic. If I understand the situation here correctly, if kube-proxy a drop rule for the invalid conntrack packets, but then the administrator set , then the result would be that conntrack would not mark some packets as invalid, and so our drop rule would just not get hit, and so our drop rule wouldn't interfere with the "better" sysctl-based solution?
Agree in general. We have done it in the past (e.g. route-localnet) and mostly it seems like a not-great idea. Also agree.
If we see this as a known bug, please consider documenting it in - even if latest Kubernetes includes a fix (we can nevertheless point that out!)
I'll fix this in the next few days.
From further googling, it seems like there have been situations in the past where people were setting that sysctl to work around bugs in conntrack, which were later fixed. I think this may be another such situation; it seems like our problem is that there is some state involving dropped or retransmitted packets which linux would cope with fine in the non-conntrack case, but which it doesn't handle in the conntrack case, unless "be_liberal" is set, and that seems like a bug.

Every time I look away from this set of issues/PRs for longer than 15 minutes I forget the exact scenario that causes the bug, but if someone could summarize it very clearly we could try dragging some kernel developers in. (The regression test in the e2e suite creates a packet with an intentionally-bad sequence number, but it's not clear to me what sort of real-world issue that's supposed to be representing.)
Thanks now that was an excellent article! Well written (not TL;DR that is), and IMO perfect technical level. If there is some collection of recommended reading for persons who want to know about K8s networking, this article should be in it.
Hi thank you for the , it's really helpful! I have some questions: , should we update this sentence accordingly ==we use the above workaround, would there any side effect for the cluster network or cluster nodes?
That says "this problem will occur if conntrack is unable to recognize a packet", but it doesn't explain the real-world scenario that causes conntrack to be unable to recognize a packet. If we go to the kernel developers and say "manually injecting random incorrect packets into a conntrack'ed connection causes it to get confused", then they'll likely respond either "don't do that then" or "use ". But if we can tell them something like "if a connection has a duplicated packet after a dropped packet, then conntrack gets confused", then that sounds a lot more like a bug in conntrack that they ought to fix, such that then everything will just work for everyone in the future without needing any DROP rules or sysctls.
So do we need to revert the drop rules? As mentioned in , it now seems that this makes sense.
There are three potential "fixes" here: this is unambiguously a conntrack bug, then we should figure out the details, and get the kernel devs to fix it. However, getting kernel bug fixes into all k8s clusters in the world takes "a long time", so even if it is a kernel bug, we should still think about the other fixes. cluster admins can use . We should advertise this better, but we feel that it would be dubious to have kube-proxy set this flag itself ().

per we could probably tweak the existing rule so that it didn't interfere with non-k8s packets ~and as suggested by the OP of this issue, we could add a similar rule to the INPUT chain (though this would need a bit of further thinking about to make sure we weren't introducing any new conflicts with non-k8s traffic)~.
We could revert the DROP rules and tell people to use the sysctl instead (and actually, probably that's a better answer for this issue than adding a new DROP rule to the INPUT chain is). But we'd need to warn users in advance if we were going to do that, since it would break existing clusters.
Totally agree, This is a better fix for this issue. If this is a kernel bug and it gets fixed someday, we can easily tell the user that you don't need to set anymore. Also, If we revert the DROP rule, which can solve the routing asymmetry issue mentioned in , which is a big fix for us Yes, We need to document it and tell the user. and In order not to break the existing cluster, Do we need a flag to decide if we need the DROP rule? I'll make it default to , and make it consistent with the current logic.
but the change to only apply the DROP rule to pod traffic should also fix that We should not require administrators to choose between "high-traffic TCP services work" and "asymmetric routing works". The current DROP rule is broken. We should either fix it to work with asymmetric routing, or else remove it and tell people who needed it to use the sysctl instead. In the latter case, we might want to make the transition via a feature gate. Then people who need asymmetric routing to work can enable the feature gate in Alpha (to make kube-proxy stop creating the DROP rule), and people who need high-traffic services to work but who didn't get the memo about using the sysctl can disable the feature gate in Beta until they have a chance to update their node configs to set the right sysctl.
Refer to , The DROP rule to pod doesn't fix our case. In our case, the pods of the external cluster subnet failed to access the pods of the cluster subnet to connect due to asymmetric routing, So the packet still matches The DROP rule to pod. Do we need a KEP for this or only a feature gate?

In general, feature gates imply KEPs (and any change that will take multiple releases to complete, where we want to make sure we don't forget about it halfway through, should get a KEP, because we have processes for tracking those). It doesn't have to be a big KEP, and some of the sections of the KEP template will be irrelevant and can be omitted, because you're not introducing a new feature. (See for example.)
We are going to break users, unfortunatly features gates does not have the effect that we are assuming, and we know that all the users have to set the "ipconntracktcpbeliberal" because eventually some connections will be broken. I do no like to set sysctl, but 'm not in favor of replacing the rule and leaving the problem to users that may not even know what a sysctl is ... agree the DROP rule is broken and breaks some esoteric but legit scenarios, but we are already setting sysctls, so I think that if we don't find any alternative to the DROP rule we should try to set the sysctl for the user by default
If we skip a KEP, make sure that the release team know to track the details (comms, docs, etc) as if it were a KEP. This is an important thing to track.
Should we at least triage this as Accepted?
/triage accepted Some alternatives DROP rule and ask users to set sysctl (I'm not in favor per my previous comment and the least surprise principle) DROP rule and we set sysctl set DROP rule if is not set the DROP rule with a flag/config so users can opt-out if don't want to set it (they can decide to use the sysctl or live with that risk) ...
I can accept KEP, but I think this may take multiple releases to complete. how about this? We remove the DROP rule Control the sysctl with a flag or a feature-gate. If it's a flag, I believe it's relatively simple and we can finish it in 1.29. In the case of feature-gate, this may take multiple releases to complete.
If sets the to true by default, we can safely remove the DROP rule
Yes, but if we were doing it that way, that would be because we considered that to be a feature rather than a bug.

We want to avoid breaking working clusters on upgrade. Maybe just having a release note saying "you may need to pass to kube-proxy" is enough. I think we agreed we don't want to change any sysctls except by explicit admin request (because that affects the behavior of non-kubernetes networking as well).
Agree, but we need to think in the users, this is a problem where the fix that solve a sublte and complex bug that may affect 100% of kubernetes users and most probably they don't know they are affected, impact a esoteric network configuration on the cluster networking, and the solution we are offering is "please read the release notes", that we know only 20% of users do ... I think that advanced users can opt-out to the DROP rule if we don't want to mess with the sysctl,
If setting was a no-brainer good idea, then the kernel would just have that behavior by default. There must be some reason why they don't, and so I worry that if we set that flag by default, then we're just going to start breaking some other subset of users. Also, if it turns out that this is all just a conntrack bug, which eventually gets fixed, then we wouldn't want to be setting the sysctl after it got fixed. But at that point it would be likely that there were some users unknowingly relying on the sysctl for some other side effect.
We could potentially have an iptables rule to count invalid conntrack packets, and expose the count as a metric
I completely and absolutely agree. we should not make that sysctl a default in kube-proxy ... what if we read the sysctl and install the DROP rule only if it is not set? Users that apply the sysctl we no longer have the DROP rule
/assign let me send a PR , I think that approach covers all cases
FWIW, my kernel conntrack informant tells me that there really aren't any bad side effects of setting it ("I don't see anything that could break by enabling this mode"), and in fact OVN sets it unconditionally on hosts it's running on. He also tells me that the bug (at least in ) is fixed in kernel ~6.2~ 6.1 ("").

So I'm not sure if that means "we should just go ahead and set the sysctl because it's harmless" or "we should just keep it like it is now and not set the sysctl since it won't be needed in the future"... (NB: if the latter we still need a docs update to go along with this PR, telling admins about the sysctl.)
If we can determine that the kernel fixes the issue in a certain version, then I think we can do this: Do not install the DROP rule.If the kernel version of the host is greater than the safe kernel version, we do not set sysctl.If less than, we have a flag to set sysctl. I'll make it default to 1. (refer to ), And let the user know that we may remove this sysctl in the future.
Exactly my thought. I made this mistake with route_localnet, and we still pay for it. Let me do some internal outreach too, see if I can get a concurring opinion?
Kernel ppl here concur: "I say : if kernel can not be fixed, set tcpbeliberal to one " "having Kubernetes set net.netfilter.nfconntracktcpbeliberal=1 always SGTM, since you want it to work on any kernel version" argues for a flag, default to true (or a hegative flag, defsult false) which seems a bit paranoid, but probably smart :) Are we all in agreement? Who wants to do the PR?
already has a PR to add to the existing set of configurable conntrack sysctls in kube-proxy () so it's just a matter of flipping the default
If the sysctl defaults to true, I think this still keeps backward compatibility, So we don't need the DROP rule anymore.
I just realized that we need this to be a flag because there are cases that run kube-proxy on environments with readonly /sys subsystem, so trying to set it always will fail will and make kube-proxy unusable for those environments
if people does not set it, they will hit the bug until they use kernel 6.1
I mean this sysctl defaults to true in kube-proxy, I can't imagine why users would need to set it to false unless the kernel version is safe (6.1). Of course, we can also document that the user that it is better not to do this.

There are cases where we CAN'T set sysctls but should not hard-fail kube-proxy. On Wed, Sep 6, 2023, 11:25 PM Cyclinder
after setting the jobs for nftables and hitting the conntrack bug continuously I think we should default the flag to true and set the sysctl nfconntracktcpbeliberal to 1 , We are already setting some sysctl and for rootless or other environment they can always opt out, this is what we do in kind
Sounds OK to me. Risk should be ~zero, right?
Yes, I assumed the risk is when to 1, we do not set it by default, so default to 0(see ). I'd be happy with it, Can I have this one?
Talked with offline, I don't have very strong arguments to make this enable by default , so let's wait for more user feedback, nftables will move to beta this release, we can revisit if we have more feedback from users
we are considering turning on by default in our kube-proxy 1.29+ deployments. Based on the recent discussion here that seems fine and the risk should "be ~zero"(), correct? And you are all even discussing turning this on by default in kube-proxy in some future version? One thing I'm a little confused on: is doing the same thing as turning on ? So by setting you're just turning on behavior that will be on by default in 6.1. Did I get that right?
Not exactly. affects a handful of situations. 6.1 only "normalized" one of them. The general intent of is something like "make conntrack work better with packets from broken TCP implementations". But the netfilter devs realized that one of the places it was being used was actually something that could happen due to totally normal TCP packet drops/retransmissions. And so in 6.1+, that particular situation is just handled correctly automatically, whether is enabled or not. My argument against this is that once we do it, we have to keep doing it forever, because people might start unwittingly depending on the other side effects of that sysctl. And particularly given that people never hit the bug anyway.

(It seems like we got more reports of the "my cluster is broken because of the DROP rule" bug than we ever got of the "my cluster is broken because of conntrack invalid packet handling" bug.) day once everyone has moved to 6.1 the bug will be gone forever ISTM that it's better to still not set it by default.
Thanks for the clarification on the 6.1 bugfix That's true and they probably will. I'm just trying to leverage some of the earlier discussion here to make a decision about the safety of flipping this to true, as it seems like the concern was not too high / basically nil.


Question: e2elog.Logf("Output of clusterapi-tester:n%v", logs) } } <ins> // getDefaultClusterIPFamily obtains the default IP family of the cluster // using the Cluster IP address of the kubernetes service created in the default namespace // This unequivocally identifies the default IP family because services are single family func getDefaultClusterIPFamily(c clientset.Interface) string { // Get the ClusterIP of the kubernetes service created in the default namespace svc, err := c.CoreV1().Services(metav1.NamespaceDefault).Get("kubernetes", metav1.GetOptions{}) if err != nil { e2elog.Failf("Failed to get kubernetes service ClusterIP: %v", err) } if utilnet.IsIPv6String(svc.Spec.ClusterIP) { return "ipv6" } return "ipv4" } </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: kind := schema.GroupKind{Group: "extensions", Kind: "DaemonSet"} framework.WaitForControlledPodsRunning(config.client, config.ns, daemonSetName, kind) <del> By("Creating a directory under discovery path") framework.Logf("creating local volume under path %q", volumePath) mkdirCmd := fmt.Sprintf("mkdir %v -m 777", volumePath) err := framework.IssueSSHCommand(mkdirCmd, framework.TestContext.Provider, config.node0) Expect(err).NotTo(HaveOccurred()) </del> By("Waiting for a PersitentVolume to be created") oldPV, err := waitForLocalPersistentVolume(config.client, volumePath) Expect(err).NotTo(HaveOccurred())

The local PV provisioner is being updated to discover only real mount points (and not directories), that resolves some other related issues. The related e2e tests using the local PV provisioner need to be updated accordingly.
/assign
/sig storage
FYI:


Question: titleDown := "%s should scale from 5 pods to 3 pods and from 3 to 1 (via %s, with scale resource: CPU)" // CPU tests via deployments <del> It(fmt.Sprintf(titleUp, "[Skipped]", kindDeployment), func() { </del> <ins> It(fmt.Sprintf(titleUp, "[Autoscaling Suite]", kindDeployment), func() { </ins> scaleUp("deployment", kindDeployment, rc, f) }) <del> It(fmt.Sprintf(titleDown, "[Skipped]", kindDeployment), func() { </del> <ins> It(fmt.Sprintf(titleDown, "[Autoscaling Suite]", kindDeployment), func() { </ins> scaleDown("deployment", kindDeployment, rc, f) })

We need to write e2e tests which combines horizontal pod autoscaling with deployments (scale sub-resourcer should point to a deployment object). The tests should be similar to The library from should be extended and allow creation of a resource consumer as a deployment (currently it is always replication controller). The new tests should create a deployment resource consumer and pass a scale-sub-resource to the deployment to autoscaler.
part of
/cc
/cc FYI requests have been made to verify behavior at scale on *this.
Yeah that sounds like a good test to run re: scale out to n node cc also


Question: `cbr-cidr` | (Optional) The minion IP address range used for the docker container bridge. `cloud` | (Optional) Which IaaS platform is used to host kubernetes, *gce*, *azure*, *aws*, *vagrant* `etcd_servers` | (Optional) Comma-delimited list of IP addresses the kube-apiserver and kubelet use to reach etcd.  Uses the IP of the first machine in the kubernetes_master role, or 127.0.0.1 on GCE. <del> `hostnamef` | (Optional) The full host name of the machine, i.e. hostname -f (only used on Azure) </del> <ins> `hostnamef` | (Optional) The full host name of the machine, i.e. uname -n </ins> `node_ip` | (Optional) The IP address to use to address this node `minion_ip` | (Optional) Mapped to the kubelet hostname_override, K8S TODO - change this name `network_mode` | (Optional) Networking model to use among nodes: *openvswitch*

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: defer utilruntime.HandleCrash() defer cc.queue.ShutDown() <del> klog.Infof("Starting certificate controller") defer klog.Infof("Shutting down certificate controller") </del> <ins> klog.Infof("Starting certificate controller %q", cc.name) defer klog.Infof("Shutting down certificate controller %q", cc.name) </ins> <del> if !cache.WaitForNamedCacheSync("certificate", stopCh, cc.csrsSynced) { </del> <ins> if !cache.WaitForNamedCacheSync(fmt.Sprintf("certificate-%s", cc.name), stopCh, cc.csrsSynced) { </ins> return }

There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


Question: }, "", "ResourceClaimStatuses okay", <ins> }, { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name: "init", }, }, Containers: []core.Container{ { Name: "nginx", }, }, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "init", Ready:       true, State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name: "init", }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyNever, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "init", Ready:       false, State: core.ContainerState{ Terminated: &core.ContainerStateTerminated{ ContainerID: "docker://numbers", Reason:      "Completed", }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, `status.initContainerStatuses[0].state: Forbidden: may not be transitioned to non-terminated state`, "init container cannot restart if RestartPolicyNever", }, { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyNever, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       true, State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyNever, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       false, State: core.ContainerState{ Terminated: &core.ContainerStateTerminated{ ContainerID: "docker://numbers", Reason:      "Completed", }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, "", "restartable init container can restart if RestartPolicyNever", }, { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyOnFailure, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       true, State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyOnFailure, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       false, State: core.ContainerState{ Terminated: &core.ContainerStateTerminated{ ContainerID: "docker://numbers", Reason:      "Completed", }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, "", "restartable init container can restart if RestartPolicyOnFailure", }, { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyAlways, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       true, State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ InitContainers: []core.Container{ { Name:          "restartable-init", RestartPolicy: &containerRestartPolicyAlways, }, }, Containers: []core.Container{ { Name: "nginx", }, }, RestartPolicy: core.RestartPolicyAlways, }, Status: core.PodStatus{ InitContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "restartable-init", Ready:       false, State: core.ContainerState{ Terminated: &core.ContainerStateTerminated{ ContainerID: "docker://numbers", Reason:      "Completed", }, }, }}, ContainerStatuses: []core.ContainerStatus{{ ContainerID: "docker://numbers", Image:       "nginx:alpine", ImageID:     "docker-pullable://nginx@sha256:d0gf00d", Name:        "nginx", Ready:       true, Started:     proto.Bool(true), State: core.ContainerState{ Running: &core.ContainerStateRunning{ StartedAt: metav1.NewTime(time.Now()), }, }, }}, }, }, "", "restartable init container can restart if RestartPolicyAlways", </ins> }, }

I created a pod with sidecar container with following manifest: watch the pod readiness. in , it's mentioned "If a readinessProbe is specified for this init container, its result will be used to determine the ready state of the Pod." it's true only for = Always, sidecar container readniess result will be used in Pod ready state, neither "Never" or "OnFailure". I tested with v1.29.0 I expect "If a readinessProbe is specified for this init container, its result will be used to determine the ready state of the Pod." also applicable to = "Never" or "OnFailure". Set = "Never" or "OnFailure". No response <details</details<details</details<details</details<details</details<detailscontainerd --version containerd v1.7.11 </details<details</details
/sig node
/remove-kind bug /kind feature I think this is a feature request, because you want to do something different than the documentation.
can you please take a look It well may be a bug /cc
/assign
confirmed that this issue is reproduced. I'll take a look. /triage accepted This happens in a container with Always containerRestartPolicy from the pod with Never or OnFailure restartPolicy. I think this is a bug. /remove-kind feature /kind bug
/priority important-soon
After a quick debug, it seems that the problem is the apiserver who is preventing the initContainerStatus from being changed from terminated to non-terminated state.
created a PR to address this.
Just a small comment on whether we need a cherry pick. This only affects specific pods with the new beta feature enabled. I don't think it is justifiable to cherry-pick this
do you think this bug is critical enough to cherry-pick. Just discussed on sidecar WG and it feels that impact may be more dangerous than I undertsood. will summarize
IIUC it doesn't seem that critical to Istio, epsecially if its high risk to backport
the fix is relatively easy, but I am operating from the assumption that everything is high risk to backport =)
I confirmed that the bug is not only limited to the sidecar container with . Kubernetes cannot update the terminated sidecar container' statuses from Job without the bugfix (regardless of the ).

The sidecar container can continually restart but its container status will be stuck in the first terminated state. It seems like a quite critical bug for me.
Updated the PR description. 1.29: 1.28:


Question: <ins> #!/usr/bin/env bash # Copyright 2023 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script checks whether the testing.init symbol is present in any # of the release binaries and fails if it finds one. This check is needed # to avoid including test libraries in production binaries as they often lack # rigorous review and sufficient testing. # Usage: `hack/verify-test-code.sh`. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" cd "${KUBE_ROOT}" RELEASE_BIN_PKGS=( "${KUBE_ROOT}/cmd/cloud-controller-manager" "${KUBE_ROOT}/cmd/kube-apiserver" "${KUBE_ROOT}/cmd/kube-controller-manager" "${KUBE_ROOT}/cmd/kube-proxy" "${KUBE_ROOT}/cmd/kube-scheduler" "${KUBE_ROOT}/cmd/kubectl" "${KUBE_ROOT}/cmd/kubectl-convert" "${KUBE_ROOT}/cmd/kubelet" ) pkgs_with_testing_import=() for file in "${RELEASE_BIN_PKGS[@]}" do if [ "$(go list -json "${file}" | jq 'any(.Deps[]; . == "testing")')" == "true" ] then pkgs_with_testing_import+=( "${file}" ) fi done if [ ${#pkgs_with_testing_import[@]} -ne 0 ]; then printf "%sn" "Testing package imported in:" for file in "${pkgs_with_testing_import[@]}"; do printf "t%sn" "${file}" done exit 1 fi exit 0 </ins>

We have many "test only" packages through out the project, containing fixtures, helpers, utilities, etc... These test libraries are not intended for use in production binaries. They often lack rigorous review and sufficient testing for use in production binaries. We should make sure they don't accidentally get pulled in. Strawman proposal, test only packages must import "testing". Blank import is sufficient: Then write a presubmit that fails if we find symbol in the release binary. This approach is trivial and fairly effective although a more sophisticated approach could print a dep chain.
/assign
I hope I will be able to contribute once this gets marked as .
/triage accepted
I think that will be enough a new script or a check in the building scripts /cc since he has experience from the providerless building effort
I got what we want to achieve here but I'm little confused about: If I execute: after adding the import then only it is giving - what do we want to achieve by executing this command exactly? I got what it is doing. I've opened a thread in slack as well:
Hello! Is there anyone working on this issue at the moment? If not and the issue it's still relevant, I would like to work on it if possible. Thank you.
I opened a PR in the meantime, if it's not needed anymore, I can close it.


Question: kube::golang::setup_env <ins> # run in module mode to match test command in readme.md export GO111MODULE=on </ins> # UPDATE_COMPATIBILITY_FIXTURE_DATA=true regenerates fixture data if needed. # -run //HEAD only runs the test cases comparing against testdata for HEAD. # We suppress the output because we are expecting to have changes. # We suppress the test failure that occurs when there are changes. <del> UPDATE_COMPATIBILITY_FIXTURE_DATA=true go test ./vendor/k8s.io/api -run //HEAD >/dev/null 2>&1 || true </del> <ins> UPDATE_COMPATIBILITY_FIXTURE_DATA=true go test k8s.io/api -run //HEAD >/dev/null 2>&1 || true </ins> # Now that we have regenerated data at HEAD, run the test without suppressing output or failures <del> go test ./vendor/k8s.io/api -run //HEAD -count=1 </del> <ins> go test k8s.io/api -run //HEAD -count=1 </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I tried rebasing an old open PR, and got merge conflicts on api testdata. I read and tried the command listed there for regenerating the test data. It failed as follows. I expected successful regeneration of the api test data. Checkout and issue the command exhibited above. Kubernetes version (use ): is at commit right nowCloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
ah, latest go versions changed the invocation to
fixed in
/triage accepted


Question: and delete the job. Job MUST be deleted successfully. release: v1.15 file: test/e2e/apps/job.go <ins> - testname: Jobs, manage lifecycle codename: '[sig-apps] Job should manage the lifecycle of a job [Conformance]' description: Attempt to create a suspended Job which MUST succeed. Attempt to patch the Job to include a new label which MUST succeed. The label MUST be found. Attempt to replace the Job to include a new annotation which MUST succeed. The annotation MUST be found. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. It MUST succeed at deleting a collection of jobs via a label selector. release: v1.25 file: test/e2e/apps/job.go </ins> - testname: Jobs, completion after task failure codename: '[sig-apps] Job should run a job to completion when tasks sometimes fail and are locally restarted [Conformance]'

[x] Test approval issue : [x] Test PR : - Needed an update on the handling of anotations 27 March 2022 [x] Two weeks soak end date : 6 April 2022 [x] Test promotion PR : [x] Update Test PR: 15 April 2022 [x] Two weeks soak end date : 4 May 2022 with start of 1.25 [x] Update Test Promotion: According to these APIsnoop queries, there are still four remaining Job endpoints which are untested. Using a number of existing e2e test practices a new has been created for this job lifecycle test. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for tracking e2e testgrid and promotion PR
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: pvs := []*v1.PersistentVolume{} pvcs := []*v1.PersistentVolumeClaim{} for i := 0; i < podLimit*volsPerPod; i++ { <ins> var ( pv      *v1.PersistentVolume pvc     *v1.PersistentVolumeClaim pvName  = fmt.Sprintf("pv-stress-%v", i) pvcName = fmt.Sprintf("pvc-stress-%v", i) ) </ins> // Don't create pvs for dynamic provisioning test if !dynamic { <del> pv := makePV(fmt.Sprintf("pv-stress-%v", i), *scName, "", "", node1) </del> <ins> if rand.Int()%2 == 0 { // static unbound pvs pv = makePV(pvName, *scName, "", "", node1) } else { // static prebound pvs pv = makePV(pvName, classImmediate, pvcName, config.ns, node1) } </ins> if pv, err := config.client.CoreV1().PersistentVolumes().Create(pv); err != nil { t.Fatalf("Failed to create PersistentVolume %q: %v", pv.Name, err) } pvs = append(pvs, pv) } <del> pvc := makePVC(fmt.Sprintf("pvc-stress-%v", i), config.ns, scName, "") </del> <ins> if pv != nil && pv.Spec.ClaimRef != nil && pv.Spec.ClaimRef.Name == pvcName { pvc = makePVC(pvcName, config.ns, &classImmediate, pv.Name) } else { pvc = makePVC(pvcName, config.ns, scName, "") } </ins> if pvc, err := config.client.CoreV1().PersistentVolumeClaims(config.ns).Create(pvc); err != nil { t.Fatalf("Failed to create PersistentVolumeClaim %q: %v", pvc.Name, err) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add prebound pvs tests cases in integration stress tests Why is this needed: See discussion here: <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
/sig storage


Question: if [[ "$STATUS" == "Succeeded" ]]; then echo "$timestamp Done." break <ins> elif [[ "$STATUS" == "Failed" ]]; then echo "$timestamp Failed." break </ins> else sleep 5 fi

Which jobs are failing: pull-kubernetes-conformance-image-test job Reason for failure: This job creates a Pod and Pod goes from being pending into "Pod status is: Failed". Please see for example the following PR to see that the job is stuck in pending state.
/assign /sig testing


Question: func findPortByName(container api.Container, portName string) (int, error) { for _, port := range container.Ports { if port.Name == portName { <del> return port.HostPort, nil </del> <ins> return port.ContainerPort, nil </ins> } } return 0, fmt.Errorf("port %s not found", portName)

When using probe, define the port by number, it works. But if the port is defined by name like below, it doesn't work. it seems should return other than .


Question: return nil } <ins> // getVmssMachineID returns the full identifier of a vmss virtual machine. func (az *Cloud) getVmssMachineID(scaleSetName, instanceID string) string { return fmt.Sprintf( vmssMachineIDTemplate, az.SubscriptionID, az.ResourceGroup, scaleSetName, instanceID) } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: } vol := vols[0] <del> tBytes, ok := newSize.AsInt64() if !ok { return fmt.Errorf("quantity %v is too great, overflows int64", newSize) </del> <ins> tBytes, err := volumehelpers.RoundUpToB(newSize) if err != nil { return err </ins> } newSizeInBytes := uint64(tBytes) if vol.Spec.Size >= newSizeInBytes {

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/