Question: } } <del> // If the kubelet config controller is available, and dynamic config is enabled, start the config and status sync loops if utilfeature.DefaultFeatureGate.Enabled(features.DynamicKubeletConfig) && len(s.DynamicConfigDir.Value()) > 0 && kubeDeps.KubeletConfigController != nil && !standaloneMode && !s.RunOnce { if err := kubeDeps.KubeletConfigController.StartSync(kubeDeps.KubeClient, kubeDeps.EventClient, string(nodeName)); err != nil { return err } } </del> if kubeDeps.Auth == nil { auth, err := BuildAuth(nodeName, kubeDeps.KubeClient, s.KubeletConfiguration) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The kubelet metrics related to , except for one, are missing from the kubelet metric endpoint. This applies to these metrics: What you expected to happen: I expect that the metrics mentioned above should be found in the endpoint on the kubelet. How to reproduce it (as minimally and precisely as possible): Run kubelet with a valid , and (defaults to true). or run the node e2e tests: Anything else we need to know?: All e2e tests related to DynamicKubeletConfig fails in master (and in v1.16): This bug/error was introduced in Environment: Kubernetes version (use ): && Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /cc logicalhan tallclair mtaufen
/cc
/assign WIP proof of concept solution to fix (make it a bit easier to understand the problem) here:
/assign
edit: I misunderstood the issue. discussed with offline, and he has an idea to restore previous behavior
That sounds perfect! He answered in the PR, so hopefully we can get this working again. I have wrapped my head around this issue for quite a few hours now, so I understand that it is kinda hard to understand by just reading this issue. Here is some more background info that hopefully makes sense.
/priority important-soon /milestone v1.17
Looks like the CI jobs are cleaning up now! ! Finally down to single digit number test failures, and away from the 30+ :smile:


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:d216b425004fcb6d8047f74e81b30e7ead55f73e73511ca53a329c358786b6c9", </del> <ins> digest = "sha256:10546d592e58d5fdb2e25d79f291b8ac62c8d3a3d83337ad7309cca766dbebce", </ins> registry = "gcr.io", repository = "google-containers/debian-hyperkube-base-amd64", <del> tag = "0.5",  # ignored, but kept here for documentation </del> <ins> tag = "0.6",  # ignored, but kept here for documentation </ins> ) docker_pull(

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods with volumeMount gitRepo volumes in 1.8.0 see the following error in kubelet logs (also in ) It is important to not that the git repo clone is done via ssh. I think what is happening is that hyperkube image now used to have command in it and now it does not have it. So the git clone command that kubelet runs fails with the above stacktrace since it cannot find . What you expected to happen: gitRepo is volume mounted. How to reproduce it (as minimally and precisely as possible): Reproducible in kubernetes 1.8.0+ hyperkube images. Anything else we need to know?: This probably is due to missing ssh binary in the new 1.8.0 hyperkube images. The commit might be an issue. Will comment further if I find otherwise. Environment: Kubernetes version (use ): =1.8.0 Cloud provider or hardware configuration**: Not Application. OS (e.g. from /etc/os-release): hyperkube base image Kernel (e.g. ): Install tools: Others:
How did you installed Kubernetes may I ask?
how does this help?
Normally, it is suggest to install kubelet as a daemon on host instead of using hyperkube container. cc you may be interested in this issue.
Need your help in identifying the fix here. I can send something for review. Suggestions are welcome.
The problem is that ssh isn't installed inside of the container. I don't know if it should be either, as the hyperkube image grows so large. If you install the openssh-client, it should work
Yes I did identify that as the problem if you see the description I mentioned. I pointed to a git commit that might have caused this. I am not sure what package installation has changed in the hyperkube container. "ssh" used to be available and now it is not. For the gitrepo feature to work it should be available.
Ah okay. If it was there before, we might want to add it back. SGTM.
Yeah, I switched the hyperkube image base from to our with some additional dependencies back. Apparently due to lack of volume + hyperkube test coverage, I've missed a number of dependencies needed.

cc
we could use dropbear as a replacement for just the ssh client?
I would prefer to use the openssh client over the dropbear client. It has wider usage and a stronger security track record IMO. The dropbear client also has somewhat different usage and configuration, so arguably using it is still a regression over the previous state. Saving a couple MB isn't worth using worse utilities, however much fun "dockerfile-golfing" is.
to fix
cherrypick to release-1.8:
This should be fixed in v1.8.3.


Question: /* <del> Copyright 2015 The Kubernetes Authors. </del> <ins> Copyright 2021 The Kubernetes Authors. </ins> Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.

What happened: follow on to Need to comment and avoid checking UID on windows. requires some research into what we should be checking so that we can add appropriate TODO. What you expected to happen: Lets also check this at the call site, Anything else we need to know?: Environment: Kubernetes version (use ): master OS (e.g: ): windows
/assign
/assign
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Hi are you working on this issue already? If not I'd like to tackle this issue :)
Hi yeah I am already working on this :)
Started branch for this here
thanks
Hey guys is there anything I can do here?
If you want to test perks patch that would be great ?
That sounds great Where do I get started?
Hey ! is there anything I can help with? and how?
I dont think there is anything left to do anything more that you think we need to do to get this merged?
Tested this with containerd as well, I think its good to test
/triage accepted


Question: : ${E2E_UP:="true"} : ${E2E_TEST:="false"} : ${E2E_DOWN:="false"} <ins> : ${ENABLE_DEPLOYMENTS:=true} </ins> NUM_MINIONS=3 ;;

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: <ins> # Create, Update & Delete Deployment with the Dynamic Package This example program demonstrates the fundamental operations for managing on [Deployment][1] resources, such as `Create`, `List`, `Update` and `Delete` using client-go's `dynamic` package. ## Typed Vs. Dynamic The code in this directory is based on a similar [example that uses Kubernetes typed client sets][2]. The typed client sets make it simple to communicate with the API server using pre-generated local API objects to achieve an RPC-like programming experience.  Typed clients uses program compilations to enforce data safety and some validation.  However, when using typed clients, programs are forced to be tightly coupled with the version and the types used. The `dynamic` package on the other hand, uses a simple type, `unstructured.Unstructured`, to represent all object values from the API server. Type `Unstructured` uses a collection of nested `map[string]interface{}` values to create an internal structure that closely resemble the REST payload from the server. The dynamic package defers all data bindings until runtime.  This means programs that use the dynamic client will not get any of the benefits of type validation until the program is running.  This may be a problem for certain types of applications that require strong data type check and validation. Being loosely coupled, however, means that programs that uses the `dynamic` package do not require recompilation when the client API changes.  The client program has more flexibility in handling updates to the API surface without knowing ahead of time what those changes are. ## Running this example Make sure you have a Kubernetes cluster and `kubectl` is configured: ``` kubectl get nodes ``` Compile this example on your workstation: ``` cd dynamic-create-update-delete-deployment go build -o ./app ``` Now, run this application on your workstation with your local kubeconfig file: ``` ./app # or specify a kubeconfig file with flag ./app -kubeconfig=$HOME/.kube/config ``` Running this command will execute the following operations on your cluster: 1. **Create Deployment:** This will create a 2 replica Deployment. Verify with `kubectl get pods`. 2. **Update Deployment:** This will update the Deployment resource created in previous step by setting the replica count to 1 and changing the container image to `nginx:1.13`. You are encouraged to inspect the retry loop that handles conflicts. Verify the new replica count and container image with `kubectl describe deployment demo`. 3. **List Deployments:** This will retrieve Deployments in the `default` namespace and print their names and replica counts. 4. **Delete Deployment:** This will delete the Deployment object and its dependent ReplicaSet resource. Verify with `kubectl get deployments`. Each step is separated by an interactive prompt. You must hit the <kbd>Return</kbd> key to proceed to the next step. You can use these prompts as a break to take time to run `kubectl` and inspect the result of the operations executed. You should see an output like the following: ``` Creating deployment... Created deployment "demo-deployment". -> Press Return key to continue. Updating deployment... Updated deployment... -> Press Return key to continue. Listing deployments in namespace "default": * demo-deployment (1 replicas) -> Press Return key to continue. Deleting deployment... Deleted deployment. ``` ## Cleanup Successfully running this program will clean the created artifacts. If you terminate the program without completing, you can clean up the created deployment with: kubectl delete deploy demo-deployment ## Troubleshooting If you are getting the following error, make sure Kubernetes version of your cluster is v1.13 or higher in `kubectl version`: panic: the server could not find the requested resource [1]: https://kubernetes.io/docs/user-guide/deployments/ [2]: ../create-update-delete-deployment </ins>

What would you like to be : Add documentation/example showing the usage of dynamic package in the client-go repo. /sig api-machinery


Question: cmd := NewCmdImage(f, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") <ins> cmd.Flags().Set("local", "true") </ins> mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper}

It tries to contact the api server for the discovery docs and fails because no api server is running. Could first look in its local cache if it's passed --local?
I think here has similar meaning as , it will not contact api-server to do real patch, but it will fetch the resource from api-server. What you mean look in it's local cache? to find the resource in local cache? I think you have specified the resource in your command line.
Use of --local flag should completely eliminate communication with api-server I have sent a PR which should fix this issue A command like kubectl set image --local -f busybox=busybox:1.26 -o yaml should load the local yaml file and print out the patched version (image should get updated to busybox:1.26) in yaml format on the screen. With the submitted patch such behavior is achived


Question: {v1.EventTypeWarning, "Unable to mount volumes for pod "recycler-for-podRecyclerFailure_default(3c9809e5-347c-11e6-a79b-3c970e965218)": timeout expired waiting for volumes to attach/mount"}, {v1.EventTypeWarning, "Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"recycler-for-podRecyclerFailure". list of unattached/unmounted"}, }, <del> expectedError: "Pod was active on the node longer than specified deadline", </del> <ins> expectedError: "failed to recycle volume: Pod was active on the node longer than specified deadline", </ins> }, { // Recycler pod gets deleted

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.


Question: glog.V(2).Infof("Opened local port %s", lp.String()) return socket, nil } <ins> // revertPorts is closing ports in replacementPortsMap but not in originalPortsMap. In other words, it only // closes the ports opened in this sync. func revertPorts(replacementPortsMap, originalPortsMap map[localPort]closeable) { for k, v := range replacementPortsMap { // Only close newly opened local ports - leave ones that were open before this update if originalPortsMap[k] == nil { glog.V(2).Infof("Closing local port %s after iptables-restore failure", k.String()) v.Close() } } } </ins>

The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.

Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:


Question: <ins> // +build linux /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dockertools import ( "fmt" "net" "path" "strconv" "testing" "github.com/golang/mock/gomock" "github.com/stretchr/testify/assert" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/client/record" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/pkg/kubelet/network" "k8s.io/kubernetes/pkg/kubelet/network/mock_network" "k8s.io/kubernetes/pkg/security/apparmor" utilstrings "k8s.io/kubernetes/pkg/util/strings" ) func TestGetSecurityOpts(t *testing.T) { const containerName = "bar" pod := func(annotations map[string]string) *v1.Pod { p := makePod("foo", &v1.PodSpec{ Containers: []v1.Container{ {Name: containerName}, }, }) p.Annotations = annotations return p } tests := []struct { msg          string pod          *v1.Pod expectedOpts []string }{{ msg:          "No security annotations", pod:          pod(nil), expectedOpts: []string{"seccomp=unconfined"}, }, { msg: "Seccomp default", pod: pod(map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: "docker/default", }), expectedOpts: nil, }, { msg: "AppArmor runtime/default", pod: pod(map[string]string{ apparmor.ContainerAnnotationKeyPrefix + containerName: apparmor.ProfileRuntimeDefault, }), expectedOpts: []string{"seccomp=unconfined"}, }, { msg: "AppArmor local profile", pod: pod(map[string]string{ apparmor.ContainerAnnotationKeyPrefix + containerName: apparmor.ProfileNamePrefix + "foo", }), expectedOpts: []string{"seccomp=unconfined", "apparmor=foo"}, }, { msg: "AppArmor and seccomp profile", pod: pod(map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: "docker/default", apparmor.ContainerAnnotationKeyPrefix + containerName:  apparmor.ProfileNamePrefix + "foo", }), expectedOpts: []string{"apparmor=foo"}, }} dm, _ := newTestDockerManagerWithVersion("1.11.1", "1.23") for i, test := range tests { securityOpts, err := dm.getSecurityOpts(test.pod, containerName) assert.NoError(t, err, "TestCase[%d]: %s", i, test.msg) opts, err := dm.fmtDockerOpts(securityOpts) assert.NoError(t, err, "TestCase[%d]: %s", i, test.msg) assert.Len(t, opts, len(test.expectedOpts), "TestCase[%d]: %s", i, test.msg) for _, opt := range test.expectedOpts { assert.Contains(t, opts, opt, "TestCase[%d]: %s", i, test.msg) } } } func TestSeccompIsUnconfinedByDefaultWithDockerV110(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.10.1", "1.22") // We want to capture events. recorder := record.NewFakeRecorder(20) dm.recorder = recorder pod := makePod("foo", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar"}, }, }) runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar.[a-f0-9]+_foo_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.Contains(t, newContainer.HostConfig.SecurityOpt, "seccomp:unconfined", "Pods with Docker versions >= 1.10 must not have seccomp disabled by default") cid := utilstrings.ShortenString(fakeDocker.Created[1], 12) assert.NoError(t, expectEvent(recorder, v1.EventTypeNormal, events.CreatedContainer, fmt.Sprintf("Created container with docker id %s; Security:[seccomp=unconfined]", cid))) } func TestUnconfinedSeccompProfileWithDockerV110(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.10.1", "1.22") pod := makePod("foo4", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar4"}, }, }) pod.Annotations = map[string]string{ v1.SeccompPodAnnotationKey: "unconfined", } runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo4_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar4.[a-f0-9]+_foo4_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.Contains(t, newContainer.HostConfig.SecurityOpt, "seccomp:unconfined", "Pods created with a secccomp annotation of unconfined should have seccomp:unconfined.") } func TestDefaultSeccompProfileWithDockerV110(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.10.1", "1.22") pod := makePod("foo1", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar1"}, }, }) pod.Annotations = map[string]string{ v1.SeccompPodAnnotationKey: "docker/default", } runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo1_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar1.[a-f0-9]+_foo1_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.NotContains(t, newContainer.HostConfig.SecurityOpt, "seccomp:unconfined", "Pods created with a secccomp annotation of docker/default should have empty security opt.") } func TestSeccompContainerAnnotationTrumpsPod(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.10.1", "1.22") pod := makePod("foo2", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar2"}, }, }) pod.Annotations = map[string]string{ v1.SeccompPodAnnotationKey:                      "unconfined", v1.SeccompContainerAnnotationKeyPrefix + "bar2": "docker/default", } runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo2_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar2.[a-f0-9]+_foo2_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.NotContains(t, newContainer.HostConfig.SecurityOpt, "seccomp:unconfined", "Container annotation should trump the pod annotation for seccomp.") } func TestSecurityOptsAreNilWithDockerV19(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.9.1", "1.21") pod := makePod("foo", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar"}, }, }) runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar.[a-f0-9]+_foo_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.NotContains(t, newContainer.HostConfig.SecurityOpt, "seccomp:unconfined", "Pods with Docker versions < 1.10 must not have seccomp disabled by default") } func TestCreateAppArmorContanier(t *testing.T) { dm, fakeDocker := newTestDockerManagerWithVersion("1.11.1", "1.23") // We want to capture events. recorder := record.NewFakeRecorder(20) dm.recorder = recorder pod := &v1.Pod{ ObjectMeta: v1.ObjectMeta{ UID:       "12345678", Name:      "foo", Namespace: "new", Annotations: map[string]string{ apparmor.ContainerAnnotationKeyPrefix + "test": apparmor.ProfileNamePrefix + "test-profile", }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "test"}, }, }, } runSyncPod(t, dm, fakeDocker, pod, nil, false) verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_test.[a-f0-9]+_foo_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() // Verify security opts. newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } securityOpts := newContainer.HostConfig.SecurityOpt assert.Contains(t, securityOpts, "apparmor=test-profile", "Container should have apparmor security opt") cid := utilstrings.ShortenString(fakeDocker.Created[1], 12) assert.NoError(t, expectEvent(recorder, v1.EventTypeNormal, events.CreatedContainer, fmt.Sprintf("Created container with docker id %s; Security:[seccomp=unconfined apparmor=test-profile]", cid))) } func TestSeccompLocalhostProfileIsLoaded(t *testing.T) { tests := []struct { annotations    map[string]string expectedSecOpt string expectedSecMsg string expectedError  string }{ { annotations: map[string]string{ v1.SeccompPodAnnotationKey: "localhost/test", }, expectedSecOpt: `seccomp={"foo":"bar"}`, expectedSecMsg: "seccomp=test(md5:21aeae45053385adebd25311f9dd9cb1)", }, { annotations: map[string]string{ v1.SeccompPodAnnotationKey: "localhost/sub/subtest", }, expectedSecOpt: `seccomp={"abc":"def"}`, expectedSecMsg: "seccomp=sub/subtest(md5:07c9bcb4db631f7ca191d6e0bca49f76)", }, { annotations: map[string]string{ v1.SeccompPodAnnotationKey: "localhost/not-existing", }, expectedError: "cannot load seccomp profile", }, } for i, test := range tests { dm, fakeDocker := newTestDockerManagerWithVersion("1.11.0", "1.23") // We want to capture events. recorder := record.NewFakeRecorder(20) dm.recorder = recorder dm.seccompProfileRoot = path.Join("fixtures", "seccomp") pod := makePod("foo2", &v1.PodSpec{ Containers: []v1.Container{ {Name: "bar2"}, }, }) pod.Annotations = test.annotations result := runSyncPod(t, dm, fakeDocker, pod, nil, test.expectedError != "") if test.expectedError != "" { assert.Contains(t, result.Error().Error(), test.expectedError) continue } verifyCalls(t, fakeDocker, []string{ // Create pod infra container. "create", "start", "inspect_container", "inspect_container", // Create container. "create", "start", "inspect_container", }) fakeDocker.Lock() if len(fakeDocker.Created) != 2 || !matchString(t, "/k8s_POD.[a-f0-9]+_foo2_new_", fakeDocker.Created[0]) || !matchString(t, "/k8s_bar2.[a-f0-9]+_foo2_new_", fakeDocker.Created[1]) { t.Errorf("unexpected containers created %v", fakeDocker.Created) } fakeDocker.Unlock() newContainer, err := fakeDocker.InspectContainer(fakeDocker.Created[1]) if err != nil { t.Fatalf("unexpected error %v", err) } assert.Contains(t, newContainer.HostConfig.SecurityOpt, test.expectedSecOpt, "The compacted seccomp json profile should be loaded.") cid := utilstrings.ShortenString(fakeDocker.Created[1], 12) assert.NoError(t, expectEvent(recorder, v1.EventTypeNormal, events.CreatedContainer, fmt.Sprintf("Created container with docker id %s; Security:[%s]", cid, test.expectedSecMsg)), "testcase %d", i) } } func TestGetPodStatusFromNetworkPlugin(t *testing.T) { cases := []struct { pod                *v1.Pod fakePodIP          string containerID        string infraContainerID   string networkStatusError error expectRunning      bool expectUnknown      bool }{ { pod: &v1.Pod{ ObjectMeta: v1.ObjectMeta{ UID:       "12345678", Name:      "foo", Namespace: "new", }, Spec: v1.PodSpec{ Containers: []v1.Container{{Name: "container"}}, }, }, fakePodIP:          "10.10.10.10", containerID:        "123", infraContainerID:   "9876", networkStatusError: nil, expectRunning:      true, expectUnknown:      false, }, { pod: &v1.Pod{ ObjectMeta: v1.ObjectMeta{ UID:       "12345678", Name:      "foo", Namespace: "new", }, Spec: v1.PodSpec{ Containers: []v1.Container{{Name: "container"}}, }, }, fakePodIP:          "", containerID:        "123", infraContainerID:   "9876", networkStatusError: fmt.Errorf("CNI plugin error"), expectRunning:      false, expectUnknown:      true, }, } for _, test := range cases { dm, fakeDocker := newTestDockerManager() ctrl := gomock.NewController(t) fnp := mock_network.NewMockNetworkPlugin(ctrl) dm.networkPlugin = fnp fakeDocker.SetFakeRunningContainers([]*FakeContainer{ { ID:      test.containerID, Name:    fmt.Sprintf("/k8s_container_%s_%s_%s_42", test.pod.Name, test.pod.Namespace, test.pod.UID), Running: true, }, { ID:      test.infraContainerID, Name:    fmt.Sprintf("/k8s_POD.%s_%s_%s_%s_42", strconv.FormatUint(generatePodInfraContainerHash(test.pod), 16), test.pod.Name, test.pod.Namespace, test.pod.UID), Running: true, }, }) fnp.EXPECT().Name().Return("someNetworkPlugin").AnyTimes() var podNetworkStatus *network.PodNetworkStatus if test.fakePodIP != "" { podNetworkStatus = &network.PodNetworkStatus{IP: net.ParseIP(test.fakePodIP)} } fnp.EXPECT().GetPodNetworkStatus(test.pod.Namespace, test.pod.Name, kubecontainer.DockerID(test.infraContainerID).ContainerID()).Return(podNetworkStatus, test.networkStatusError) podStatus, err := dm.GetPodStatus(test.pod.UID, test.pod.Name, test.pod.Namespace) if err != nil { t.Fatal(err) } if podStatus.IP != test.fakePodIP { t.Errorf("Got wrong ip, expected %v, got %v", test.fakePodIP, podStatus.IP) } expectedStatesCount := 0 var expectedState kubecontainer.ContainerState if test.expectRunning { expectedState = kubecontainer.ContainerStateRunning } else if test.expectUnknown { expectedState = kubecontainer.ContainerStateUnknown } else { t.Errorf("Some state has to be expected") } for _, containerStatus := range podStatus.ContainerStatuses { if containerStatus.State == expectedState { expectedStatesCount++ } } if expectedStatesCount < 1 { t.Errorf("Invalid count of containers with expected state") } } } </ins>

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: // move all pods to active queue when we were trying to schedule them q.MoveAllToActiveQueue() <del> moveReqChan := make(chan struct{}) var wg sync.WaitGroup wg.Add(totalNum - 1) // mark pods[1] ~ pods[totalNum-1] as unschedulable, fire goroutines to add them back later </del> <ins> oldCycle := q.SchedulingCycle() firstPod, _ := q.Pop() if !reflect.DeepEqual(&expectedPods[0], firstPod) { t.Errorf("Unexpected pod. Expected: %v, got: %v", &expectedPods[0], firstPod) } // mark pods[1] ~ pods[totalNum-1] as unschedulable and add them back </ins> for i := 1; i < totalNum; i++ { unschedulablePod := expectedPods[i].DeepCopy() unschedulablePod.Status = v1.PodStatus{

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: continue } informerPodUID := volumehelper.GetUniquePodName(informerPod) <del> // Check whether the unique idenfier of the pod from dsw matches the one retrived from pod informer </del> <ins> // Check whether the unique identifier of the pod from dsw matches the one retrieved from pod informer </ins> if informerPodUID == dswPodUID { glog.V(10).Infof( "Verified pod %q (UID %q) from dsw exists in pod informer.", dswPodKey, dswPodUID)

Problem: An attachable volume referenced by a pod via a PVC object (instead of directly) is not detached if the pod is deleted via namespace deletion (i.e. deleting the pod's namespace instead of the pod object directly). Repro steps: a new volume a new namespace. - PV and PVC objects: - PV/PVC objects are in bound state - pod object that references the volume via PVC - pod gets into running state without issue (i.e. volume is attached) - the namespace - for pod to terminate if volume remains attached after a few minutes Expected: Volume is detached. Actual: Volume remains attached indefinitely. <detailsis: And is: </detailsWorkarounds: the pod object directly before deleting the namespace. In step 7 above do instead If the attachable volume is referenced by a pod using a PVC object, and the pod object is deleted directly (instead of the namespace first), the volume is correctly detached. volume directly in pod (without a PVC object). If the attachable volume is referenced by a pod directly (without a PVC object), namespace deletion results in the volume being correctly detached.
Root Cause: event for pod fails on Attach/Detach controller When a volume referenced via PVC is deleted by namespace, the PV/PVC objects are deleted before the pod object, therefore when the delete event is processed by the attach/detach controller, it fails because it cannot dereference the PVC object. was to ensure volumes are still detached even if a delete event is missed (or fails), however that has a bug in it (the value of is not checked): Proposed Fixes: attach/detach controller to cache PVC/PV so even if it is deleted, it can be recovered on pod deletion events. bug in to check so that it can delete pods even if the delete event is missed (or fails).


Question: By("deleting host1Pod") framework.ExpectNoError(podClient.Delete(host1Pod.Name, &api.DeleteOptions{}), "Failed to delete host1Pod") <ins> By("Test completed successfully, waiting for PD to safely detach") waitForPDDetach(diskName, host0Name) waitForPDDetach(diskName, host1Name) </ins> return })

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: when listed by its label selector. release: v1.19 file: test/e2e/common/node/podtemplates.go <ins> - testname: PodTemplate, replace codename: '[sig-node] PodTemplates should replace a pod template [Conformance]' description: Attempt to create a PodTemplate which MUST succeed. Attempt to replace the PodTemplate to include a new annotation which MUST succeed. The annotation MUST be found in the new PodTemplate. release: v1.24 file: test/e2e/common/node/podtemplates.go </ins> - testname: PodTemplate lifecycle codename: '[sig-node] PodTemplates should run the lifecycle of PodTemplates [Conformance]' description: Attempt to create a PodTemplate. Patch the created PodTemplate. Fetching

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 14 March 2022 [X] test promotion pr : According to this APIsnoop query, there is still one remaining PodTemplate endpoint which is untested. Using a number of existing e2e test practices a new has been created for pod templates. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, } } <ins> // Ensure a key is in the store before returning (or timeout w/ error) func WaitForStoreUpdate(store util.FederatedReadOnlyStore, clusterName, key string, timeout time.Duration) error { retryInterval := 100 * time.Millisecond err := wait.PollImmediate(retryInterval, timeout, func() (bool, error) { _, found, err := store.GetByKey(clusterName, key) return found, err }) return err } </ins>

https://k8s- Failed: TestNamespaceController {namespace} Previous issues for this test:
GetObjectFromChan() is timing out (20s) and returning nil causing the panic. Happens in TestSecretController (and probably TestIngressController) controller as well. https://k8s-
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
I have a fix for this in
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
/cc OWNERS of :
I think this is also related to Issue .
fix is pending


Question: // This is very low impact. The NodePort range is intentionally obscure, and unlikely to actually collide with real Services. // This only affects UDP connections, which are not common. // See issue: https://github.com/kubernetes/kubernetes/issues/49881 <del> isIPv6 := utilproxy.IsIPv6(svcInfo.clusterIP) </del> err := utilproxy.ClearUDPConntrackForPort(proxier.exec, lp.Port, isIPv6) if err != nil { glog.Errorf("Failed to clear udp conntrack for port %d, error: %v", lp.Port, err)

/kind bug /sig network If I set up a service with a NodePort and set , and try to access it as it works. If I try to access it as , it triggers a martian source error - the packet gets forwarded as 127.0.0.1. We need some masquerade in there. Applies to iptables mode. Maybe ipvs mode, not sure?
If we set for NodePort, we don't do any masq for packets currently. I have tested that masq will resolve the issue in iptables mode, but my question is "do we still need to preserve source ip of packets for nodeport when externalTrafficPolicy=Local"? If we do masq, we will lose the source IP, sadly.
BTW, with the fix of , is reachable The iptables created by is Any comments?
this is not entirely true. key "externalTrafficPolicy=Local" needs to be a solution to the problem of obtaining the real IP of the client, but this is not happening. If we do masq, we will not lose the source IP, because now he doesn't work (version 1.8.1) the problem with the inaccessibility of the actual source IP () in version 1.8.1 is still relevant and is a very big problem for deployment in kubernetes most web services on a production. besides, in my cluster key "externalTrafficPolicy : false" leads to a hollow stop traffic to the pods.
Without masq, nodeport for localhost when set "externalTrafficPolicy=Local" won't work. If service is not reachable, say "how to preserve source IP" won't make sense. Listen, masq is used for someone who want to visit nodeport via localhost when set "externalTrafficPolicy=Local" and don't require preserving source IP. If source IP is really important to you, you can just ignore the .
I do not understand what do you mean "you can just ignore the 127.0.0.1". could you describe in detail how to configure the service to get the real source IP?

type service, externalTrafficPolicy=[Local | Cluster] type service, externalTrafficPolicy=Local, but don't try to access type service, externalTrafficPolicy=Local
i wrote above, NodePort type service with externalTrafficPolicy=Local blocking the transmission of traffic to the pod. I'm trying to connect to node_ip:port. maybe this is a bug? I could demonstrate this.
oh, sorry )) i'm connecting to wrong node )) yes, now traffic is gone. but ip is not source! with "externalTrafficPolicy=Cluster" i getting ip 10.1.29.21 (node ip) with "externalTrafficPolicy=Local" i getting ip 10.4.62.1 (pods network ip - not real source ip)
You probably should check the iptables applied in your host, is there any masq rules created by other processes other than kube-proxy? It preserves the source IP of client - I double checked it. BTW, you can create another thread for discussing your problems. I feel like this issue is orthogonal with your question.
Will setting ClusterIP type service and externalTrafficPolicy=Local for a service to a DaemonSet, route traffic only to the daemonSet pod in that node when i access the service using DaemonSetServiceIP?
no. local service does not work for cluster IP


Question: fs.BoolVar(&c.EnableDebuggingHandlers, "enable-debugging-handlers", c.EnableDebuggingHandlers, "Enables server endpoints for log collection and local running of containers and commands") fs.BoolVar(&c.EnableContentionProfiling, "contention-profiling", c.EnableContentionProfiling, "Enable lock contention profiling, if profiling is enabled") fs.Int32Var(&c.HealthzPort, "healthz-port", c.HealthzPort, "The port of the localhost healthz endpoint (set to 0 to disable)") <del> fs.Var(utilflag.IPVar{Val: &c.HealthzBindAddress}, "healthz-bind-address", "The IP address for the healthz server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces)") </del> <ins> fs.Var(utilflag.IPVar{Val: &c.HealthzBindAddress}, "healthz-bind-address", "The IP address for the healthz server to serve on (set to '0.0.0.0' or '::' for listening in all interfaces and IP families)") </ins> fs.Int32Var(&c.OOMScoreAdj, "oom-score-adj", c.OOMScoreAdj, "The oom-score-adj value for kubelet process. Values must be within the range [-1000, 1000]") fs.StringVar(&c.ClusterDomain, "cluster-domain", c.ClusterDomain, "Domain for this cluster.  If set, kubelet will configure all containers to search this domain in addition to the host's search domains")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are flag descriptions where is stated that the unspecified address behaves differently depending on the IP family: However, that's not the behavior implemented in and , clarified in the golang documentation What you expected to happen: Fix the comment to state the reality, i.e How to reproduce it (as minimally and precisely as possible): From the root of the kubernetes repo Anything else we need to know?: xref Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind cleanup /help wanted /good-first-issue /priority important-longterm /cc
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign /remove-help
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
For the flag in kube-proxy, is used if the address is not specified and can't be resolved by hostname, so I suggest to remove this comment: 0.0.0.0::` for all IPv6 interfaces'
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I am very much in favor of this. Being correct and precise in these flags is valuable. To convince myself you were right, I wrote this program: And then I ran it for various inputs: In these flags and docs, we should be clear that if you are specifying a v6 addr and port you have to use syntax. Or maybe we should just audit and fix ALL such places to support and document that "nothing" is how you specify all interfaces. E.g. --bind-address=":80" is more obvious than either "0.0.0.0:80" or "[::]:80".

Even if those syntaxes work, let's just make "" the norm?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: informerFactory := informers.NewSharedInformerFactory(client, controller.NoResyncPeriodFunc()) spc := newFakeStatefulPodControl(informerFactory.Core().V1().Pods(), informerFactory.Apps().V1beta1().StatefulSets()) ssu := newFakeStatefulSetStatusUpdater(informerFactory.Apps().V1beta1().StatefulSets()) <del> ssc := defaultStatefulSetControl{spc, ssu, history.NewFakeHistory(informerFactory.Apps().V1beta1().ControllerRevisions())} </del> <ins> recorder := record.NewFakeRecorder(10) ssc := defaultStatefulSetControl{spc, ssu, history.NewFakeHistory(informerFactory.Apps().V1beta1().ControllerRevisions()), recorder} </ins> stop := make(chan struct{}) defer close(stop)

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: ) const regexDescribe = "Describe|KubeDescribe|SIGDescribe" <del> const regexContext = "Context" </del> <ins> const regexContext = "^Context$" </ins> type visitor struct { <del> FileSet      *token.FileSet lastDescribe describe cMap         ast.CommentMap </del> <ins> FileSet   *token.FileSet describes []describe cMap      ast.CommentMap </ins> //list of all the conformance tests in the path tests []conformanceData } //describe contains text associated with ginkgo describe container type describe struct { <ins> rparen      token.Pos </ins> text        string lastContext context }

What happened: tried promoting a test with a tag in its full test description. The hierarchy of ginko-relevant AST's was . The tag was a in the top level . The full test description only came from , so the tag was missed. What you expected to happen: The conformance test list generator should have logged and ERROR and failed a test on the PR, calling out Disruptive as an invalid tag How to reproduce it (as minimally and precisely as possible): Check out locally and follow the directions in to generate the conformance list. The invalid test is in the list. Anything else we need to know?: Is there a function we can use to get the full test description from Ginkgo instead of rolling our own? If not, we should probably be maintaining a stack rather than just the last FYI who ran into this FYI who wrote the code to keep track of last
/sig architecture /area conformance
/milestone v1.16
/priority important-soon
/assign
GitHub didn't allow me to assign the following users: devaips. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: match: minimum_go_version=go([0-9]+.[0-9]+) - name: "registry.k8s.io/kube-cross: dependents" <del> version: v1.30.0-go1.22-bullseye.0 </del> <ins> version: v1.30.0-go1.22.0-bullseye.0 </ins> refPaths: - path: build/build-image/cross/VERSION

https://prow.ppc64le- Tests are not getting executed due to improper go version. It has been failing since past 4 days. No response No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to be failing due to improper version addition in dependency files where is instead of at some places. I will raise a PR for it.
/assign
fixing that


Question: --output-version="": Output the formatted object with the given version (default api-version). --overwrite=false: If true, allow labels to be overwritten, otherwise reject label updates that overwrite existing labels. --resource-version="": If non-empty, the labels update will only succeed if this is the current resource-version for the object. <del> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. </del> <ins> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile.  The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview] </ins> Global Flags: --alsologtostderr=false: log to standard error as well as files

periodic-conformance-main-k8s-main Kubernetes e2e suite.SynchronizedBeforeSuite 07/17 23:52PDT We saw a similar issue in , which was fixed by upgrading the control plane OS from Ubuntu 18.04 to 22.04. /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig node It seems to be related to SWAP.
There is a bug in containerd that is fixed in v1.6.14. The test here is using v1.6.8. Kubernetes v1.27 supported containerd versions: 1.7.0+, 1.6.15+
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still failing after is merged. Have we tried bumping the control plane OS image to ubuntu 22.04, like ?
Nevermind, the latest testgrid job might have been triggered before the PR was merged. I am observing this and will close this once I see the job passes.
Containerd version is already v1.6.21 in 's log There maybe still other problems.
can fix a similar failure in .
/close The failure of the swap is fixed by . The current failure is a flake that is mentioned in
Closing this issue. <detailsIn response to : /close The current failure is a flake that is mentioned in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" "os" <ins> "strings" </ins> "k8s.io/kubernetes/federation/pkg/kubefed" _ "k8s.io/kubernetes/pkg/client/metrics/prometheus" // for client metric registration

/kind bug What happened: Federation set up using kubefed uses invalid reference format, breaking the Federation control plane. What you expected to happen: That it works. How to reproduce it (as minimally and precisely as possible): Try to set up a control plane using kubefed. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0+", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-02T09:48:25Z", GoVersion:"go1.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.6-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-27T21:21:34Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: GCP OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): Irrelevant Install tools: GKE
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig federation /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc might be able to help.
/assign
Assigning to you from the build cop rotation.
the fix for this issue is identified and fixed in . But it will take a while to be available on stable releases. Meanwhile you can use flag to to pass specific hyperkube version.


Question: ## Purpose <del> Implement KEP 32: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/0032-create-a-k8s-io-component-repo.md </del> <ins> Implement KEP 32: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/wgs/0032-create-a-k8s-io-component-repo.md </ins> The proposal is essentially about refactoring the Kubernetes core package structure in a way that all core components may share common code around: - ComponentConfig implementation

What happened: Looks like KEP organization changed and the component-base kep in the readme doesn't link anymore. What you expected to happen: Link to go to the right document. How to reproduce it (as minimally and precisely as possible): click link - Anything else we need to know?: n/a Environment: n/a
/sig cluster-lifecycle /sig api-machinery
/assign EDIT: oops looks like the PR is already up


Question: } } <ins> func TestPodEligibleToPreemptOthers(t *testing.T) { tests := []struct { name                string pod                 *v1.Pod pods                []*v1.Pod nodes               []string nominatedNodeStatus *framework.Status expected            bool }{ { name:                "Pod with nominated node", pod:                 st.MakePod().Name("p_with_nominated_node").UID("p").Priority(highPriority).NominatedNodeName("node1").Obj(), pods:                []*v1.Pod{st.MakePod().Name("p1").UID("p1").Priority(lowPriority).Node("node1").Terminating().Obj()}, nodes:               []string{"node1"}, nominatedNodeStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, tainttoleration.ErrReasonNotMatch), expected:            true, }, { name:                "Pod with nominated node, but without nominated node status", pod:                 st.MakePod().Name("p_without_status").UID("p").Priority(highPriority).NominatedNodeName("node1").Obj(), pods:                []*v1.Pod{st.MakePod().Name("p1").UID("p1").Priority(lowPriority).Node("node1").Terminating().Obj()}, nodes:               []string{"node1"}, nominatedNodeStatus: nil, expected:            false, }, { name:                "Pod without nominated node", pod:                 st.MakePod().Name("p_without_nominated_node").UID("p").Priority(highPriority).Obj(), pods:                []*v1.Pod{}, nodes:               []string{}, nominatedNodeStatus: nil, expected:            true, }, { name:                "Pod with 'PreemptNever' preemption policy", pod:                 st.MakePod().Name("p_with_preempt_never_policy").UID("p").Priority(highPriority).PreemptionPolicy(v1.PreemptNever).Obj(), pods:                []*v1.Pod{}, nodes:               []string{}, nominatedNodeStatus: nil, expected:            false, }, } for _, test := range tests { var nodes []*v1.Node for _, n := range test.nodes { nodes = append(nodes, st.MakeNode().Name(n).Obj()) } snapshot := internalcache.NewSnapshot(test.pods, nodes) if got := podEligibleToPreemptOthers(test.pod, snapshot.NodeInfos(), test.nominatedNodeStatus); got != test.expected { t.Errorf("expected %t, got %t for pod: %s", test.expected, got, test.pod.Name) } } } </ins> func TestNodesWherePreemptionMightHelp(t *testing.T) { // Prepare 4 nodes names. nodeNames := []string{"node1", "node2", "node3", "node4"}

What happened: When a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage, the scheduler doesn't consider that pod for preemption even there are pods can be preempted on another node. What you expected to happen: When the pod's nominated node is considered as 'UnschedulableAndUnresolvable ' (For example, the taint being by those conditions, ), the scheduler should try to preemption on other nodes. How to reproduce it (as minimally and precisely as possible): Create low priority pod, which schedule to nodeA Create mid priority pod, which schedule to nodeB Create high priority pod, which preempt the low priority pod on nodeA, and set high priority pod's nominated node as nodeA The nodeA's kubelet is down, the node life cycle controller taint the nodeA with '' (or taint with disk ) before low priority pod finishes the termination. The high priority pod is keeping as pending status. The scheduler doesn't try to preempt the mid-priority pod on nodeB Anything else we need to know?: Environment: Kubernetes version (use ): 18.0 Cloud provider or hardware configuration: baremetal OS (e.g: ): ubuntu1804 Kernel (e.g. ): 0-46-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
/assign
This seems to be a duplicate of .
actually, I found this case after review . I file this issue since I feel it should be another case. The want to cleanup the nomnated node for cluster autoscaler. This is case need to allow the pod to preempt on other nodes again even it has a nomanted node. But I may understand incorrectly, please mark it duplicated if I'm wrong. Thanks for help anyway. :)
If the nominatedNodeName can be cleared properly in any case, this issue will no longer exist. From this point of view, I'd say this issue is a sub-case of .
BTW: in your description, if "The nodeA is down", how it comes out "a Pod's nominated node is considered as UnschedulableAndUnresolvable in the filter stage"? nodeA should NOT have existed in the result of Filter(), right?

(if the nodeA is just being shutdown, we still treat it as a normal node)
If the NodeA is down, the node life cycle controller will add taint '' to the node, then the TaintToleration filter will return 'UnschedulableAndUnresolvable '. It also means that it isn't only about nodeA is down, there are other conditions in this ' to return 'UnschedulableAndUnresolvable'. Beyond the TaintToleration filter, other filters may return 'UnschedulableAndUnresolvable' also, as my understanding, that means that node won't be available for that pod anymore, then we should try other nodes for preemption.
then we should better describe it as "being applied with a Taint", which is more general. When you say "node is down", it usually refers to the node is deleted physically.
I got it, thanks again! let me correct it
Also it's different from . Let's not mention "node is down" in the description to avoid confusion.
thanks, done :)


Question: $ kubectl proxy --port=8011 --www=./local/www/ // Run a proxy to kubernetes apiserver, changing the api prefix to k8s-api <del> // This makes e.g. the pods api available at localhost:8011/k8s-api/v1beta1/pods/ $ kubectl proxy --api-prefix=k8s-api </del> <ins> // This makes e.g. the pods api available at localhost:8011/k8s-api/v1beta3/pods/ $ kubectl proxy --api-prefix=/k8s-api </ins> .fi .RE

Without , works just fine, With '--api-prefix=xxx-api', it stops work: My version: could you take a look? Thanks.
--api-prefix was in . could you take a look? This appears to be completely broken, so we should fix it or rip it out.
Reading the code and I find the way to fix it. You need to use , note the slashes. I will fix the man page.
What's an example command line that works?
then
Ah, the default flag value has leading and trailing slashes: The example in the documentation is wrong: Re-opening for doc fix.
The trailing slash is not necessary. and duplicate each other.


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package vsphere_volume import ( "os" "path" "testing" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" utiltesting "k8s.io/client-go/util/testing" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" ) var ( testVolumePath = "volPath1" testGlobalPath = "plugins/kubernetes.io/vsphere-volume/volumeDevices/volPath1" testPodPath    = "pods/poduid/volumeDevices/kubernetes.io~vsphere-volume" ) func TestGetVolumeSpecFromGlobalMapPath(t *testing.T) { // make our test path for fake GlobalMapPath // /tmp symbolized our pluginDir // /tmp/testGlobalPathXXXXX/plugins/kubernetes.io/vsphere-volume/volumeDevices/ tmpVDir, err := utiltesting.MkTmpdir("vsphereBlockVolume") if err != nil { t.Fatalf("cant' make a temp dir: %s", err) } // deferred clean up defer os.RemoveAll(tmpVDir) expectedGlobalPath := path.Join(tmpVDir, testGlobalPath) // Bad Path badspec, err := getVolumeSpecFromGlobalMapPath("") if badspec != nil || err == nil { t.Errorf("Expected not to get spec from GlobalMapPath but did") } // Good Path spec, err := getVolumeSpecFromGlobalMapPath(expectedGlobalPath) if spec == nil || err != nil { t.Fatalf("Failed to get spec from GlobalMapPath: %s", err) } if spec.PersistentVolume.Spec.VsphereVolume.VolumePath != testVolumePath { t.Fatalf("Invalid volumePath from GlobalMapPath spec: %s", spec.PersistentVolume.Spec.VsphereVolume.VolumePath) } block := v1.PersistentVolumeBlock specMode := spec.PersistentVolume.Spec.VolumeMode if &specMode == nil { t.Errorf("Invalid volumeMode from GlobalMapPath spec: %v expected: %v", &specMode, block) } if *specMode != block { t.Errorf("Invalid volumeMode from GlobalMapPath spec: %v expected: %v", *specMode, block) } } func TestGetPodAndPluginMapPaths(t *testing.T) { tmpVDir, err := utiltesting.MkTmpdir("vsphereBlockVolume") if err != nil { t.Fatalf("cant' make a temp dir: %s", err) } // deferred clean up defer os.RemoveAll(tmpVDir) expectedGlobalPath := path.Join(tmpVDir, testGlobalPath) expectedPodPath := path.Join(tmpVDir, testPodPath) spec := getTestVolume(true) // block volume pluginMgr := volume.VolumePluginMgr{} pluginMgr.InitPlugins(ProbeVolumePlugins(), nil, volumetest.NewFakeVolumeHost(tmpVDir, nil, nil)) plugin, err := pluginMgr.FindMapperPluginByName(vsphereVolumePluginName) if err != nil { os.RemoveAll(tmpVDir) t.Fatalf("Can't find the plugin by name: %q", vsphereVolumePluginName) } if plugin.GetPluginName() != vsphereVolumePluginName { t.Fatalf("Wrong name: %s", plugin.GetPluginName()) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: types.UID("poduid"), }, } mapper, err := plugin.NewBlockVolumeMapper(spec, pod, volume.VolumeOptions{}) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mapper == nil { t.Fatalf("Got a nil Mounter") } // GetGlobalMapPath globalMapPath, err := mapper.GetGlobalMapPath(spec) if err != nil || len(globalMapPath) == 0 { t.Fatalf("Invalid GlobalMapPath from spec: %s", spec.PersistentVolume.Spec.VsphereVolume.VolumePath) } if globalMapPath != expectedGlobalPath { t.Errorf("Failed to get GlobalMapPath: %s %s", globalMapPath, expectedGlobalPath) } // GetPodDeviceMapPath devicePath, volumeName := mapper.GetPodDeviceMapPath() if devicePath != expectedPodPath { t.Errorf("Got unexpected pod path: %s, expected %s", devicePath, expectedPodPath) } if volumeName != testVolumePath { t.Errorf("Got unexpected volNamne: %s, expected %s", volumeName, testVolumePath) } } func getTestVolume(isBlock bool) *volume.Spec { pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: testVolumePath, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ VsphereVolume: &v1.VsphereVirtualDiskVolumeSource{ VolumePath: testVolumePath, }, }, }, } if isBlock { blockMode := v1.PersistentVolumeBlock pv.Spec.VolumeMode = &blockMode } return volume.NewSpecFromPersistentVolume(pv, true) } </ins>

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: verbs:     "get,watch,list", resources: "pods,pods", expectedClusterRole: &rbac.ClusterRole{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "ClusterRole"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: clusterRoleName, },


Question: srcs = [":package-srcs"], tags = ["automanaged"], ) <ins> go_test( name = "go_default_test", srcs = ["meta_proxier_test.go"], embed = [":go_default_library"], deps = ["//staging/src/k8s.io/api/core/v1:go_default_library"], ) </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Setup a kubernetes cluster with dual stack enabled, set kube-proxy to "ipvs" proxy mode, many warnings are flooding in kube-proxy pod log, W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:26. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) What you expected to happen: No such error logs How to reproduce it (as minimally and precisely as possible): Here is my setup step, I'm using a single master node setup, kubeadm init --pod-network-cidr=192.168.0.0/16,fc00::/112 --service-cidr=10.96.0.0/16,fd00::/112 --feature-gates=IPv6DualStack=true Change controller-manager cidr configuration --node-cidr-mask-size-ipv6=120 install calico 3.6.4 with dual stack enabled kube-proxy proxy mode set to "ipvs" featureGates: IPv6DualStack: true healthzBindAddress: "" hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 0s kind: KubeProxyConfiguration metricsBindAddress: "" mode: "ipvs" Anything else we need to know?

: Linux Redhat 7.7 Environment: Kubernetes version (use ):v1.17.3 Cloud provider or hardware configuration: OS (e.g: ): Redhat 7.7 Kernel (e.g. ): 3.10.0-1062.4.1.el7.x8664 Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico 3.6.4 Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved those warnings are harmless because those objects doesn't have endpoints addresses, thus no IP family, as you can see below: however, I think that if there are no endpoint addresses we should skip instead of logging, what do you think ?
It seems these printouts can be caused by headless services, please see;
Can endpoints support dual stack? From my understanding, it cannot support dual stack in v1.17. Why kube-proxy will check another ipFamily which is not configured? Here is the endpoints info under default ns from my cluster,
Another observation about dual stack supporting is I don't see ipv6 interface address in node status. Here is my interface configuration, and here is pod configuration, I see kubelet has already set ipv6 network in podCIDRs
The is reported by if you specify . Currently kubelet does not report both ipv4 and ipv6 addresses, that is not implemented yet. But public clouds do not let report the node but defines them themselves. At least set both addresses. So it depends on your environment.
endpoints on services with ClusterIP services are single family, with the same IPFamily than the ClusterIP. However, you can create endpoints manually, not associated to a service, or to a headless service (ClusterIP=None) with a mix of families. You opened the issue asking why do you have these errors logs If you check the endpoints of the services mentioned in the logs there are no addresses, hence the kube-proxy logs a WARNING because is not able to identify the IP family. Technically the warning is correct, but I think that if there are no addresses we shouldn't log the warning.

In order to not convert this issue in a placeholder for dual stack let's try to stick to the topic, for more general things we have a #k8s-dual-stack channel in slack.
Ok, I understand what you mentioned. From the implementation, this log behavior is correct. The other problem is kube-proxy is constantly logging the same error.
I think (please somebody correct me if I'm wrong) this is because the endpoints are checked periodically, per the control-loop / reconcile paradigm that kubernetes uses, so it tries to add those endpoints periodically
nodeIPs are only dual stack if the cloud-controller-manager updates it, on bare metal or without cloud-controller-manager you only see one address. I tried cloud-controller-manager in my openstack cluster and can get ipv4/ipv6 interface.
Thanks, I will create another issue for document.


Question: }) } } <ins> func TestNumFeasibleNodesToFind(t *testing.T) { tests := []struct { name                     string percentageOfNodesToScore int32 numAllNodes              int32 wantNumNodes             int32 }{ { name:         "not set percentageOfNodesToScore and nodes number not more than 50", numAllNodes:  10, wantNumNodes: 10, }, { name:                     "set percentageOfNodesToScore and nodes number not more than 50", percentageOfNodesToScore: 40, numAllNodes:              10, wantNumNodes:             10, }, { name:         "not set percentageOfNodesToScore and nodes number more than 50", numAllNodes:  1000, wantNumNodes: 420, }, { name:                     "set percentageOfNodesToScore and nodes number more than 50", percentageOfNodesToScore: 40, numAllNodes:              1000, wantNumNodes:             400, }, { name:         "not set percentageOfNodesToScore and nodes number more than 50*125", numAllNodes:  6000, wantNumNodes: 300, }, { name:                     "set percentageOfNodesToScore and nodes number more than 50*125", percentageOfNodesToScore: 40, numAllNodes:              6000, wantNumNodes:             2400, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { g := &genericScheduler{ percentageOfNodesToScore: tt.percentageOfNodesToScore, } if gotNumNodes := g.numFeasibleNodesToFind(tt.numAllNodes); gotNumNodes != tt.wantNumNodes { t.Errorf("genericScheduler.numFeasibleNodesToFind() = %v, want %v", gotNumNodes, tt.wantNumNodes) } }) } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Kubernetes scheduler has that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. The scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration. The default value of this config option is 50 (for 50%). We should let the scheduler determine this value based on the size of the cluster when this config option is not provided. I'd propose this formula for finding the percentage: . This is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster. The scheduler already has a lower bound of 50 nodes per scheduling cycle. As a part of this change, we should also add a 5% lower bound for the formula (If the result of the formula is below 5, we should return 5). Why is this needed: To improve scheduler performance in large clusters. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig scheduling
/assign /remove-help
/assign wgliang
Thanks


Question: oldTsPath := filepath.Join(w.targetDir, oldTsDir) var pathsToRemove sets.String <ins> shouldWrite := true </ins> // if there was no old version, there's nothing to remove if len(oldTsDir) != 0 { // (3)

When kubelet is creating configmap volume for a pod, it finished writing dir and made symlink to . At the same time, the kubelet is restarted, unfortunately the configmap item symlink is not created. The restarted kubelet will try to configmap, . Kubelet should create absent symlink for configmap volume. can hack kubelet to help reproduce it: sleep some time before ` kubelet and start official kubelet will skip to create symlink for UserVisibleFiles No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig storage This seems like an issue for sig storage. Can someone take a look and confirm?
Can you please add the version details as well?
/remove-sig node
/assign


Question: apiVersion: apps/v1 kind: DaemonSet metadata: <del> name: npd-v0.8.0 </del> <ins> name: npd-v0.8.5 </ins> namespace: kube-system labels: k8s-app: node-problem-detector <del> version: v0.8.0 </del> <ins> version: v0.8.5 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: node-problem-detector <del> version: v0.8.0 </del> <ins> version: v0.8.5 </ins> template: metadata: labels: k8s-app: node-problem-detector <del> version: v0.8.0 </del> <ins> version: v0.8.5 </ins> kubernetes.io/cluster-service: "true" spec: containers: - name: node-problem-detector <del> image: k8s.gcr.io/node-problem-detector:v0.8.0 </del> <ins> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.5 </ins> command: - "/bin/sh" - "-c"

What happened: In a cluster with NPD addon installed and Linux 5.1+ as the kernel OS of the nodes OOMs are not being captured by the NPD. What you expected to happen: NPD should inform about OOMs by emitting appropriate K8s events. How to reproduce it (as minimally and precisely as possible): Create a K8s cluster with NPD addon and Linux 5.1+ installed on the nodes. Schedule a pod which leaks memory. Try to list events with field after an OOM occurs to see there's none. Anything else we need to know?: Fix is already merged into newest (v0.8.5) NPD release: We're currently waiting for the release of Docker image of NPD v0.8.5 - we can proceed with fixing this in k/k afterwards. After it gets merged to the master branch, it should be then cherry-picked to older releases as well. Environment: Any Kubernetes version with NPD addon installed and Linux as the node OS with the kernel version 5.1 or newer.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability /sig node I opened a fixing PR in but it's on hold since the Docker image of NPD v0.8.5 doesn't exist yet.


Question: type Server struct { host    HostInterface updates chan<- interface{} <del> handler http.Handler </del> <ins> mux     *http.ServeMux </ins> } <del> func ListenAndServeKubeletServer(host HostInterface, updates chan<- interface{}, delegate http.Handler, address string, port uint) { </del> <ins> // ListenAndServeKubeletServer initializes a server to respond to HTTP network requests on the Kubelet func ListenAndServeKubeletServer(host HostInterface, updates chan<- interface{}, address string, port uint) { </ins> glog.Infof("Starting to listen on %s:%d", address, port) <del> handler := Server{ host:    host, updates: updates, handler: delegate, } </del> <ins> handler := NewServer(host, updates) </ins> s := &http.Server{ Addr:           net.JoinHostPort(address, strconv.FormatUint(uint64(port), 10)), Handler:        &handler,

The Kubelet REST end-point has turned into a complicated switch-case code block that is harder to maintain overtime. The Kubelet should move to a http.ServeMux pattern and register handler functions to a pattern.
I am working on a PR to submit in next couple of days that will make this change, so feel free to assign to me if possible.
Tagging because I asked for this in one of his PRs
This one is also under my radar. If no one had time, feel free to assign it to me too.
I have a branch in progress. Feel free to offer feedback on it when I submit. Sent from my iPhone


Question: } // mount it globalPDPath := util.MakeGlobalPDName(*b.fcDisk) <ins> if err := os.MkdirAll(globalPDPath, 0750); err != nil { return devicePath, fmt.Errorf("fc: failed to mkdir %s, error", globalPDPath) } </ins> noMnt, err := b.mounter.IsLikelyNotMountPoint(globalPDPath) if err != nil { return devicePath, fmt.Errorf("Heuristic determination of mount point failed:%v", err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod creation doesn't finish due to the following error. If the directory doesn't exist it will be created instead of returning error. What you expected to happen: Pod with FC volume works properly. How to reproduce it (as minimally and precisely as possible): Clean up under the "/var/lib/kubelet/plugins" Create a Pod with FC volume Environment: Kubernetes version (use ):Install tools: local-up- /cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage


Question: "testing" "time" <ins> sdktrace "go.opentelemetry.io/otel/sdk/trace" "go.opentelemetry.io/otel/sdk/trace/tracetest" </ins> oteltrace "go.opentelemetry.io/otel/trace" "github.com/golang/mock/gomock"

spans are not being parented to other spans - propagation of context seems broken. More context: spans should be parented to other spans enable kubelet tracing at v1.28 and hook up a tracing backend. No response <detailsv1.28 at least </details<detailsNot sure this matters. </details<details</details<details</details<details</details<details</details
/sig instrumentation /assign
/triage accepted
!
repro steps:
/assign Thanks for helping to look into this! Feel free to post the findings here you shared with me on slack
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks! I will post this ASAP
The broken issue is due to the pr , which try to propagate context brought by pod worker to syncPod function in order to let the syncPod can be interrupted in time by other asynchronous routines, for example eviction. But in the line we did not pass the context which contain the parent span and trace into function, so the trace is broken. So I want to fix this by passing the into , which is also desired in comment But passing the ctx into syncPod may have something other to do as described in comments, so I tried to fix as comments advised.
cc


Question: for i := 0; i < diff; i++ { go func(ix int) { defer wait.Done() <del> if err := rm.podControl.DeletePod(rc.Namespace, filteredPods[ix].Name); err != nil { </del> <ins> if err := rm.podControl.DeletePod(rc.Namespace, filteredPods[ix].Name, rc); err != nil { </ins> // Decrement the expected number of deletes because the informer won't observe this deletion glog.V(2).Infof("Failed deletion, decrementing expectations for controller %q/%q", rc.Namespace, rc.Name) rm.expectations.DeletionObserved(rcKey)

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc


Question: t.Errorf("%s: Did not get expected path for exec request", test.name) return } <ins> if strings.Count(ex.url.RawQuery, "container=bar") != 1 { t.Errorf("%s: Did not get expected container query param for exec request", test.name) return } </ins> if ex.method != "POST" { t.Errorf("%s: Did not get method for exec request: %s", test.name, ex.method) }

<!-- --What happened: when use command, we found multi query param in the request, because we set the param twice in What you expected to happen: How to reproduce it (as minimally and precisely as possible): use command and set the log level to Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
It seems a duplicate 'container' param are set, thanks for your report. I will fix it later.


Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package conversion import ( "reflect" "testing" "github.com/google/gofuzz" ) func TestDeepCopy(t *testing.T) { semantic := EqualitiesOrDie() f := fuzz.New().NilChance(.5).NumElements(0, 100) table := []interface{}{ map[string]string{}, int(5), "hello world", struct { A, B, C struct { D map[string]int } X []int Y []byte }{}, } for _, obj := range table { obj2, err := DeepCopy(obj) if err != nil { t.Errorf("Error: couldn't copy %#v", obj) continue } if e, a := obj, obj2; !semantic.DeepEqual(e, a) { t.Errorf("expected %#vngot %#v", e, a) } obj3 := reflect.New(reflect.TypeOf(obj)).Interface() f.Fuzz(obj3) obj4, err := DeepCopy(obj3) if err != nil { t.Errorf("Error: couldn't copy %#v", obj) continue } if e, a := obj3, obj4; !semantic.DeepEqual(e, a) { t.Errorf("expected %#vngot %#v", e, a) } f.Fuzz(obj3) } } func copyOrDie(t *testing.T, in interface{}) interface{} { out, err := DeepCopy(in) if err != nil { t.Fatalf("DeepCopy failed: %#q: %v", in, err) } return out } func TestDeepCopySliceSeparate(t *testing.T) { x := []int{5} y := copyOrDie(t, x).([]int) x[0] = 3 if y[0] == 3 { t.Errorf("deep copy wasn't deep: %#q %#q", x, y) } } func TestDeepCopyArraySeparate(t *testing.T) { x := [1]int{5} y := copyOrDie(t, x).([1]int) x[0] = 3 if y[0] == 3 { t.Errorf("deep copy wasn't deep: %#q %#q", x, y) } } func TestDeepCopyMapSeparate(t *testing.T) { x := map[string]int{"foo": 5} y := copyOrDie(t, x).(map[string]int) x["foo"] = 3 if y["foo"] == 3 { t.Errorf("deep copy wasn't deep: %#q %#q", x, y) } } func TestDeepCopyPointerSeparate(t *testing.T) { z := 5 x := &z y := copyOrDie(t, x).(*int) *x = 3 if *y == 3 { t.Errorf("deep copy wasn't deep: %#q %#q", x, y) } } </ins>

E.g., it calls Convert() for every byte of a slice of uint8. E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting []v1.NamedExtension E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting []uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23.

5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 ... thousands of these.
There's an easy fix for this :/


Question: }, timeout: 3 * time.Second, <del> expectedErr: wait.ErrWaitTimeout.Error(), </del> <ins> expectedErr: "timed out waiting for the condition on theresource/name-foo", </ins> validateActions: func(t *testing.T, actions []clienttesting.Action) { if len(actions) != 4 { t.Fatal(spew.Sdump(actions))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Make sure reset happens on patch requests // wait until new webhook is called if err := wait.PollImmediate(time.Millisecond*5, wait.ForeverTestTimeout, func() (bool, error) { <ins> defer recordedWarnings.Reset() </ins> pod, err = client.CoreV1().Pods("default").Patch(context.TODO(), invalidManagedFieldsMarkerFixture.Name, types.JSONPatchType, []byte("[]"), metav1.PatchOptions{}) if err != nil { return false, err

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-integration TestMutatingWebhookResetsInvalidManagedFields Not sure. Example failures from recent PRs: (PR: ) (PR: ) /sig api-machinery /kind flake
There's even a few failures in the post-submit master integration tests job: (ignore the test/integration/job failures, I have an open PR to fix that).
cc
Thanks! /cc /wg api-expression
Oh no, what did I do :-/ I'll have a look. Might be that the admission webhook is not up quick enough. /assign
/triage accepted
I tried this locally and it passes, so maybe I can try stress testing this
Would increasing the timelimit here give more time to the admissionhook to come up ?
I think so. Looked at it a few days ago, I'm not sure how much we should increase it. I don't want to slow down the tests a lot more if we can avoid it. Do we know if the other admission tests (duplicate owner ref) have the same issue?
Can we make the webhook blocking so that the request will wait for it to be ready? Can we make sure that it's up-and-running somehow before performing the actual test? Having an arbitrary "wait" before we start the test seems wrong.
The poll and timeout are fine as is. lastErr is only getting set, never unset. If validateManagedFieldsAndDecode returns nil, lastErr should get cleared.
Hmm.. actually, the one failing is the Update request, after the Patch request has already been checked. The patch request passing means the webhook was effective and working, the update request should work consistently the first time. If the patch request really is ensuring the webhook was installed, then a timeout or retry on the update request is not the right approach.
But isnt the patch request alos failing to report the adequate warnings
this continues to be the flakiest test in the integration job in by a wide margin
So sorry for letting this slip for so long :-/ I'll work on it right when I get back home in an hour.