Question: limits = quota.Max(limits, pod.Spec.InitContainers[i].Resources.Limits) } <ins> if feature.DefaultFeatureGate.Enabled(features.PodOverhead) { requests = quota.Add(requests, pod.Spec.Overhead) limits = quota.Add(limits, pod.Spec.Overhead) } </ins> result = quota.Add(result, podComputeUsageHelper(requests, limits)) return result, nil }

assume that: cluster with only one node with 4G allocatable memory Define test runtimeclass with PodOverhead Define resourceQuota test under test namespace with 4Gi memory limit Create Pod with test runtimeClass under test namespace podOverhead resource not counted into resourceQuota used info podOverhead resource counted into node requests info try to create one more pod with 2G memory requests, resourceQuota check passed but failed to schedule because it requests 2G+200M memory podOverhead resource should be counted into resourceQuota used info to keep resourceQuota check and schedule check be consistent. Kubernetes version 1.16
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node
ref:
-- can you verify that you have the feature gate enabled for pod overhead for api-server, as well as kubelet and scheduler? Example:
I made a cluster with the following command, using crio as a container runtime( is not available with Docker) And the issue was reproduced.
overhead feature enabled, the overhead resource has been counted into requests during scheduling but not counted into resourcequota
Currently, PodOverhead is defined as a fixed resource. this value looks like the average usage value and being into requests resource during scheduling. type Overhead struct { PodFixed *ResourceList } My use case is kata_container, the VMM related processes consume much overhead resource when pod running with high workload. A small overhead(small cgroup limits) will trigger oom but a big value result that big overhead requests take into account during scheduling. So how about define PodOverhead including limits and requests like that in Pod resources definition. It could be helpful resources: limits: memory: 1Gi requests: memory: 200M
CC: FYI


Question: tokenJsonBlob, err := credentialprovider.ReadUrl(metadataToken, g.Client, metadataHeader) if err != nil { <ins> glog.Errorf("while reading access token endpoint: %v", err) </ins> return cfg } email, err := credentialprovider.ReadUrl(metadataEmail, g.Client, metadataHeader) if err != nil { <ins> glog.Errorf("while reading email endpoint: %v", err) </ins> return cfg }

It looks like we always try and hit the metadata server even if we aren't on GCE. We then print an error out. This should either be silent (assume after X attempts that we aren't on GCE?) or should only be activated with an option. Email thread: !topic/google-containers/w-ei5Xs6K0Y
cc:
FYI, the way the credentialprovider logic is set up, providers are asked once per keyring creation whether they are enabled. If they are enabled, then they are asked to provide their contribution to .dockercfg each time the lazy keyring is accessed. For efficiency, there is a caching provider, which composes with heavier providers and stores their .dockercfg contribution for a predetermined TTL. What was happening here was that "Enabled()" for our GCE-metadata implementations was logging about errors fetching "http://metadata", where the entire "Enabled()" check is "err == nil".
I believe that this was fixed by


Question: package windows import ( <del> "fmt" "io" "net/http" "os" </del> <ins> "time" appsv1 "k8s.io/api/apps/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/pkg/controller/deployment/util" "k8s.io/kubernetes/test/e2e/framework" </ins> ) <del> // downloadFile saves a remote URL to a local temp file, and returns its path. // It's the caller's responsibility to clean up the temp file when done. func downloadFile(url string) (string, error) { response, err := http.Get(url) if err != nil { return "", fmt.Errorf("unable to download from %q: %w", url, err) } defer response.Body.Close() tempFile, err := os.CreateTemp("", "") if err != nil { return "", fmt.Errorf("unable to create temp file: %w", err) } defer tempFile.Close() _, err = io.Copy(tempFile, response.Body) return tempFile.Name(), err </del> <ins> // waits for a deployment to be created and the desired replicas // are updated and available, and no old pods are running. func waitForDeployment(getDeploymentFunc func() (*appsv1.Deployment, error), interval, timeout time.Duration) error { return wait.PollImmediate(interval, timeout, func() (bool, error) { deployment, err := getDeploymentFunc() if err != nil { if apierrors.IsNotFound(err) { framework.Logf("deployment not found, continue waiting: %s", err) return false, nil } framework.Logf("error while deploying, error %s", err) return false, err } framework.Logf("deployment status %s", &deployment.Status) return util.DeploymentComplete(deployment, &deployment.Status), nil }) </ins> }

Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing


Question: cd "${KUBE_TARGET}" <del> if [ "$1" != "" ]; then go test -race $KUBE_TIMEOUT $KUBE_COVER -coverprofile=tmp.out "$KUBE_GO_PACKAGE/$1" "${@:2}" </del> <ins> while getopts "i:" opt ; do case $opt in i) iterations=$OPTARG ;; ?) echo "Invalid argument -$OPTARG" exit 1 ;; :) echo "Option -$OPTARG <value>" exit 1 ;; esac done shift $((OPTIND - 1)) if [[ -n "${iterations}" ]]; then echo "Running ${iterations} times" if [[ -n "$1" ]]; then pkg=$KUBE_GO_PACKAGE/$1 fi rm -f *.test # build a test binary echo "${pkg}" go test -c -race ${KUBE_TIMEOUT} "${pkg}" # keep going, even if there are failures pass=0 count=0 for i in $(seq 1 ${iterations}); do for test_binary in *.test; do if "./${test_binary}"; then ((pass++)) fi ((count++)) done done 2>&1 echo "${pass}" / "${count}" passing if [[ ${pass} != ${count} ]]; then exit 1 else exit 0 fi fi if [[ -n "$1" ]]; then go test -race ${KUBE_TIMEOUT} ${KUBE_COVER} -coverprofile=tmp.out "${KUBE_GO_PACKAGE}/$1" "${@:2}" </ins> exit 0 fi <del> find_test_dirs | xargs go test -race -timeout 30s $KUBE_COVER "${@:2}" </del> <ins> find_test_dirs | xargs go test -race -timeout 30s ${KUBE_COVER} "${@:2}" </ins>

Since 8th JUL, the test cases don't show up on testgrid: It's not clear from testgrid, which runs pass and which failed atm 2022-07-08 Since 8th JUL, the build-log and junit artifacts looks different. Before, the junit file was split into parts: and it contains only a data (name, time, skipped*, ...), E.g: From , there's a single file, which is and contains all the and informations, which is a duplication of build logs. E.g.: Looking at the difference in kubernetes/kubernetes source code - there were number of changes to: ginkgo, klog and traces - all could be related: cc - I see you've made number of commits; would you be able to help? No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/milestone 1.25
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.25 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.25
I think that this is related to the ginkgov2 migration, the reporters in ginkgo has changed. Do you have a link to the code where the scale test run the e2e commands?
The test is defined here: We use kubekins.

My guess is that the old "junit shards" we see were generated by different ginkgo processes/threads (number of junit files matches set in our test) and mentions that we the old parameter "--parallel" no longer works. Probably we need to migrate kubekins to use instead of "--parallel".
hmm, that seems a kubetest flag ,
I just checked kubetest code and it looks like the kubetest's --ginkgo-parallel flag gets propagated to --nodes ginkgo flag and it seems to work fine: The most recent test () has in build log.
it seems is using --nodes for parallel I think that now it aggregates the reports, also running in parallel
besides that, all the other jobs are reporting a single junit xml file, is this the real problem? do you have more ideas on why this file is not being interpreted by testgrid?
My guess is that 600MiB of junit exceeds some size limit in used in testgrid and for that reason it isn't interpreted by testgrid.
re: I think this envvar needs to be changed from: to Ref:
that doesn't look related, AFAIK the migration is about an API and that is a kubetest variable, we are passing the flag correctly. I think that the junit generation is a better lead, in k/k there is tooling to strip the file to avoid issues with large sizes it seems kubetest dumps it entirely someone is up to adapt kubetest to either split the xml generated or prune the large messages in k/k?
sorry, I just notice this, I am trying to answer some questions here if possible. as has said, Ginkgo V2 merged all results into a single junit file, the file is named as "", pls see:
it's only for golang's parallel flag, we shouldn't use that as it's not accepted by Ginkgov2, here is an example, is fine.
I don't think so, pls see:
+1 on the pruning and adapt with new junit file.
In deck's logs I see: So at least for deck the reason for not showing junit is too large file. I think testgrid implements similar logic probably.
is there any way to keep a simplified junit report instead of dump everything in that file? as what Ginkgo V1 did?

Deck is using limit of 200M:
the solution seems to adapt kubetest I can try to take a shot after code freeze
This is really something that we shouldn't have to do in kubetest :/ (consider this also impacts tools like sonobuoy), and we're generally not accepting more complexity in that tool
Attached v1 junit report and v2 junit report as a reference, v2 has embedded into each testcases even the testcase is skipped, this make the file for v2 is larger than v1. v1, v2 Just run one testcases, and v1 has 21145L, and v2 has 28258L
I don't know what's the below line bringing us anything when the testcase is skipped. is that by design?
might be easiest if we trim / split / ... in a custom reporter or on exit in , otherwise we have to consider kubetest, kubetest2, sonobuoy, etc.
how can we inject something like this then?
hey all, when running in parallel ginkgo v2 now merges the individual reports generated by each parallel process into one composite process. there is no way to turn this off (v1's behavior was a shortcut at the time and somewhat ugly and confusing). v2 also merges junit reports from multiple suites into a single file. this can be turned off with in v2 i also took a closer look at what few official-seeming junit specs exist and updated the reporter to match. currently there is no mechanism to control that default behavior. you can, however, build a custom reporter to do some filtering/processing in-suite and for a project of k8s scope that might make the most sense. for example - a reporter that modifies/filters the report object before manually calling manually. I'd be happy to share more detail/help if that approach is preferred.
How can we trigger this CI: ci-kubernetes-e2e-gce-scale-correctness?
I'm not sure if you can easily, as the project is constantly used by correctness and performance tests. Or at least I don't know how to run it - I've just wrote down in this issue to what I've observed.

FYI this is CI job that is being run every day: It requires 5k nodes cluster to run and we have 1 gcp project shared with other scalability tests that can handle it so there is no easy way to manually trigger it.
I have tried to split, but split the 600+ M xml file would be time consuming. I am not sure whether that job rely on the , if so, we might consider to add a custom reporter for that. For what I observed for each jobs triggered regularly for a PR, the size of report is less than 50M. The gap between 50 and 600 are huge, poor background for me to get more info on that job. Talking about the custom reporter, shall the testgrid or Spyglass smart enough to fetch the result from two different junit xml file? one is trimmed and the other one is original.
/cc for thoughts.
I think we should try to reduce the amount of information that gets stored in the single JUnit file. We also should ensure that we only store a single output stream (). That addresses the main problem ("too much information"). Splitting into different files feels like a poor workaround because eventually they need to be merged again in memory when dealing with the entire job run. I don't think we need the original, more complete JUnit file.
I am afraid someone need the original one, since the original one holds the single of truth if anyone wants to check the details on the test execution. +1 for couple of reasons, said that the split by the the number of parallelism is just a workaround at that time, v2 sound like did the right things, but this doesn't work for testgrid, . split is time consuming and not trivial to implement.
Here is the comparison after this pr: original one: - trimmed one: - more than 90% off. similarly, we will eventually get around 60 M for the job ?
Spyglass seems work, here is the , and is not impacted when the test failed. while the testcase passed or skipped, Spyglass doesn't provide the link to the or , so the Spyglass should be fine. Spyglass reads the trimmed xml file btw, not sure how it selects the file though.
checked all testcase status should be back. , the size now is about 2.5M.

But one thing should be noted is that, name of the testcases has been changed a little bit which is caused by the new format junit xml. Here is one example, In V1, the testcase named as: "Kubernetes e2e suite.[sig-storage] Volumes NFSv4 should be mountable for NFSv4" now has been updated to: "Kubernetes e2e suite.[It] [sig-storage] Volumes NFSv4 should be mountable for NFSv4" pls note the string of "[It]" that is . So, you can only check the status of "Kubernetes e2e suite.[It] [sig-storage] Volumes NFSv4 should be mountable for NFSv4" now. We can somehow modify the report again to trim the string "[It]", this should be easy, but I am not quite sure whether we need to make everything the same as before. If you are okay with the new test name, I think we can close this issue, otherwise we can update the name as well. thoughts?
I personally think it's okay, k8s just deliver this file while how the downstream tools consume those data is not predictable. If the "IT" string to the testcase is not necessary then it should be a bug from Ginkgo, and should be fixed in Ginkgo, in this case, we can filter out the string as a temporary workaround.
Another diff I found, All skipped testcase won't shown in testgrid, e.g. "[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" the new format is <testcase name="[It] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" classname="Kubernetes e2e suite" status="skipped" time="0.

"<skipped message="skipped - Driver nfs doesn't support Block -- skipping"</skipped</testcasepls note the attribute of "status="skipped" , it was, <testcase name="[sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (block volmode)] multiVolume [Slow] should access to two volumes with different volume mode and retain data across pod recreation on different node" classname="Kubernetes e2e suite" time="0"<skipped</skipped</testcaseCan testgrid or some tool grep the "status" to get the result instead? do you know someone familiar with the tool can help to update the tool a little bit?
I see , it seems the new reporting format breaks testgrid, can you help us to recover the functionality in testgrid?
the release 1.25 bug triage Lead. I'm reaching out to see the status of this issue and if we are still targeting the current release. Also, is it a release blocker? Can you please update the priority as well?
FWIW I think this is implemented here (junit in GCS =testgrid)
I don't think this is a release blocker, the change needed in Kubernetes has been merged, meanwhile, I pushed a pr in testgrid to close this
Thanks for replying back, I'll move it to 1.26 then
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.26 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.26
/close this issue has been resolved, I ca see that the test cases are showing up now, since the different fixes to the e2e framework. The remaining part is to show the skip test cases in testgrid, but that is another issue, better not to add more things here
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: OffendingContainersUsageKey = "offending_containers_usage" // StarvedResourceKey is the key for the starved resource in eviction event annotations StarvedResourceKey = "starved_resource" <ins> // thresholdMetMessageFmt is the message for evictions due to resource pressure. thresholdMetMessageFmt = "Threshold quantity: %v, available: %v. " </ins> ) var (

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: . "github.com/onsi/gomega" ) <ins> func LaunchNetTestPodPerNodeNew(f *Framework, nodes *api.NodeList, name string) []string { podNames := []string{} totalPods := len(nodes.Items) Expect(totalPods).NotTo(Equal(0)) for _, node := range nodes.Items { pod, err := f.Client.Pods(f.Namespace.Name).Create(&api.Pod{ ObjectMeta: api.ObjectMeta{ GenerateName: name + "-", Labels: map[string]string{ "name": name, }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "webserver", Image: "gcr.io/google_containers/nettest:1.4", Args: []string{ "-service=" + name, //peers >= totalPods should be asserted by the container. //the nettest container finds peers by looking up list of svc endpoints. fmt.Sprintf("-peers=%d", totalPods), "-namespace=" + f.Namespace.Name}, Ports: []api.ContainerPort{{ContainerPort: 8080}}, }, }, Host:          node.Name, RestartPolicy: api.RestartPolicyNever, }, }) Expect(err).NotTo(HaveOccurred()) Logf("Created pod %s on node %s", pod.ObjectMeta.Name, node.Name) podNames = append(podNames, pod.ObjectMeta.Name) } return podNames } var _ = Describe("NetworkingNew", func() { f := NewFramework("nettestnew") var svcname = "nettest" BeforeEach(func() { //Assert basic external connectivity. //Since this is not really a test of kubernetes in any way, we //leave it as a pre-test assertion, rather than a Ginko test. By("Executing a successful http request from the external internet") resp, err := http.Get("http://google.com") if err != nil { Failf("Unable to connect/talk to the internet: %v", err) } if resp.StatusCode != http.StatusOK { Failf("Unexpected error code, expected 200, got, %v (%v)", resp.StatusCode, resp) } }) // First test because it has no dependencies on variables created later on. It("should provide unchanging, static URL paths for kubernetes api services.", func() { tests := []struct { path string }{ {path: "/validate"}, {path: "/healthz"}, // TODO: test proxy links here } for _, test := range tests { By(fmt.Sprintf("testing: %s", test.path)) data, err := f.Client.RESTClient.Get(). Namespace(f.Namespace.Name). AbsPath(test.path). DoRaw() if err != nil { Failf("Failed: %vnBody: %s", err, string(data)) } } }) //Now we can proceed with the test. It("should function for intra-pod communication", func() { if testContext.Provider == "vagrant" { By("Skipping test which is broken for vagrant (See https://github.com/GoogleCloudPlatform/kubernetes/issues/3580)") return } By(fmt.Sprintf("Creating a service named %q in namespace %q", svcname, f.Namespace.Name)) svc, err := f.Client.Services(f.Namespace.Name).Create(&api.Service{ ObjectMeta: api.ObjectMeta{ Name: svcname, Labels: map[string]string{ "name": svcname, }, }, Spec: api.ServiceSpec{ Ports: []api.ServicePort{{ Protocol:   "TCP", Port:       8080, TargetPort: util.NewIntOrStringFromInt(8080), }}, Selector: map[string]string{ "name": svcname, }, }, }) if err != nil { Failf("unable to create test service named [%s] %v", svc.Name, err) } // Clean up service defer func() { defer GinkgoRecover() By("Cleaning up the service") if err = f.Client.Services(f.Namespace.Name).Delete(svc.Name); err != nil { Failf("unable to delete svc %v: %v", svc.Name, err) } }() By("Creating a webserver (pending) pod on each node") nodes, err := f.Client.Nodes().List(labels.Everything(), fields.Everything()) if err != nil { Failf("Failed to list nodes: %v", err) } podNames := LaunchNetTestPodPerNodeNew(f, nodes, svcname) // Clean up the pods defer func() { defer GinkgoRecover() By("Cleaning up the webserver pods") for _, podName := range podNames { if err = f.Client.Pods(f.Namespace.Name).Delete(podName, nil); err != nil { Logf("Failed to delete pod %s: %v", podName, err) } } }() By("Waiting for the webserver pods to transition to Running state") for _, podName := range podNames { err = f.WaitForPodRunning(podName) Expect(err).NotTo(HaveOccurred()) } By("Waiting for connectivity to be verified") passed := false //once response OK, evaluate response body for pass/fail. var body []byte getDetails := func() ([]byte, error) { return f.Client.Get(). Namespace(f.Namespace.Name). Prefix("proxy"). Resource("services"). Name(svc.Name). Suffix("read"). DoRaw() } getStatus := func() ([]byte, error) { return f.Client.Get(). Namespace(f.Namespace.Name). Prefix("proxy"). Resource("services"). Name(svc.Name). Suffix("status"). DoRaw() } timeout := time.Now().Add(2 * time.Minute) for i := 0; !passed && timeout.After(time.Now()); i++ { time.Sleep(2 * time.Second) Logf("About to make a proxy status call") start := time.Now() body, err = getStatus() Logf("Proxy status call returned in %v", time.Since(start)) if err != nil { Logf("Attempt %v: service/pod still starting. (error: '%v')", i, err) continue } // Finally, we pass/fail the test based on if the container's response body, as to wether or not it was able to find peers. switch { case string(body) == "pass": Logf("Passed on attempt %v. Cleaning up.", i) passed = true case string(body) == "running": Logf("Attempt %v: test still running", i) case string(body) == "fail": if body, err = getDetails(); err != nil { Failf("Failed on attempt %v. Cleaning up. Error reading details: %v", i, err) } else { Failf("Failed on attempt %v. Cleaning up. Details:n%s", i, string(body)) } case strings.Contains(string(body), "no endpoints available"): Logf("Attempt %v: waiting on service/endpoints", i) default: Logf("Unexpected response:n%s", body) } } if !passed { if body, err = getDetails(); err != nil { Failf("Timed out. Cleaning up. Error reading details: %v", err) } else { Failf("Timed out. Cleaning up. Details:n%s", string(body)) } } Expect(string(body)).To(Equal("pass")) }) }) </ins> func LaunchNetTestPodPerNode(f *Framework, nodes *api.NodeList, name string) []string { podNames := []string{}

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.


Question: options := []string{} if readOnly { options = append(options, "ro") <ins> } else { // as per https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting options = append(options, "discard") </ins> } if notMnt { diskMounter := &mount.SafeFormatAndMount{Interface: mounter, Runner: exec.New()}

GCE PD docs so unused blocks can be freed back into the cloud and performance gains realized. Rummaging through the source of the GCE PD and mounter stuff in Kube, it does not appear that that's being done (the old shell script version of safeformatand_mount does, but I don't think that's used). I haven't confirmed that it's the case at runtime for sure, though, although I will be able to shortly.
I LGTMed the PR and then realized it needed more work. Reverting it and assigning it to ciwang. We need to make sure that we only use the "discard" mount option when the fs supports it. Assigning to ciwang for now. If anyone else wants to work on it feel free to grab it.
Assigning to This fell through the cracks as ciwang was working in other areas. This did not make 1.4 either moving to next-candidate.
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: if err != nil { return result, err } <del> trace.Step("Prioritizing done") </del> metrics.SchedulingAlgorithmPriorityEvaluationDuration.Observe(metrics.SinceInSeconds(startPriorityEvalTime)) metrics.DeprecatedSchedulingAlgorithmPriorityEvaluationDuration.Observe(metrics.SinceInMicroseconds(startPriorityEvalTime)) metrics.SchedulingLatency.WithLabelValues(metrics.PriorityEvaluation).Observe(metrics.SinceInSeconds(startPriorityEvalTime)) metrics.DeprecatedSchedulingLatency.WithLabelValues(metrics.PriorityEvaluation).Observe(metrics.SinceInSeconds(startPriorityEvalTime)) host, err := g.selectHost(priorityList) <del> trace.Step("Selecting host done") </del> <ins> trace.Step("Prioritizing done") </ins> return ScheduleResult{ SuggestedHost:  host, EvaluatedNodes: len(filteredNodes) + len(failedPredicateMap) + len(filteredNodesStatuses),

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We have enabled in genericscheduler. However, the steps being traced seem outdated. In particular, the "snapshot" is expensive and should be traced. Other steps can be traced as well wherever appropriate, e.g., PreFilter. Why is this needed: Improve granularity of tracing. /sig scheduling
/assign


Question: }) } } <ins> func Test_RaceTerminateRSList(t *testing.T) { ipvs := &utilipvstest.FakeIPVS{} gracefulTerminationManager := NewGracefulTerminationManager(ipvs) go func() { for i := 1; i <= 10; i++ { for j := 1; i <= 100; j++ { gracefulTerminationManager.rsList.add(makeListItem(i, j)) } } }() if !gracefulTerminationManager.rsList.flushList(gracefulTerminationManager.deleteRsFunc) { t.Error("failed to flush entries") } } func makeListItem(i, j int) *listItem { vs := fmt.Sprintf("%d.%d.%d.%d", 1, 1, i, i) rs := fmt.Sprintf("%d.%d.%d.%d", 1, 1, i, j) return &listItem{ VirtualServer: &utilipvs.VirtualServer{ Address:  netutils.ParseIPSloppy(vs), Protocol: "tcp", Port:     uint16(80), }, RealServer: &utilipvs.RealServer{ Address: netutils.ParseIPSloppy(rs), Port:    uint16(80), }, } } </ins>

! no panic Kubernetes version (use ): k8s 1.16
/sig Network
Could you fix this? I think the function needs a lock while being executed.
/triage accepted /good-first-issue todo: add a lock around the body of this function: Similar to
/assign
Oh I see, thank you for letting me know, I'll keep that in mind going forward. :)
Anyone actually on this? It seems like a good fix to have for 23 and maybe backport
I'll have a try.
There is a reproducer here
Oops, I just peeked at the time of the pull request and not realized that you've got such deep conversation.
/remove-good-first-issue /unassign
/assign
hi ! someone work on this now? I would like to try it.
Yes, working on it currently.
are you still planning to give this a shot?
Is there no one working on this? If so, I'd submit a PR for it this week. :)


Question: import ( "fmt" "reflect" <ins> "strconv" </ins> "testing" "github.com/google/gofuzz"

In Go you cannot take the address of a because the map implementation hides that from you. Our conversion library assumes that any reflected map value can have Addr() taken, which causes a panic when you try to do the following for PodInfo conversion: The conversion code tries to call Addr() on the reflect.Value (the map's value, ContainerStatus value object) which fails because it cannot have an address. I worked around it by registering a conversion function on *PodInfo and then manually converting the map.
Oh you mean the reflection code can't addr that? ok sounds like a bug. Non-reflection code can certainly do it:
That playground is taking the address of a value before it's in the struct. You can't take the address of m ----- Original Message -----
Right, because go does magical c++ style references there. But I thought the map copying code handled that case specially already?
I think I see what you're talking about. I'll make a test.
I repro'd. Will fix.
It was actually the source value that caused the trouble. Thanks for the report!
Sorry, should have been clearer. ----- Original Message -----
Note that this map violates our API conventions. It should be a list of named subobjects.
Yeah this field has always been problematic. It's still a bug regardless-- could have triggered copying other maps.


Question: dswp.actualStateOfWorld.MarkRemountRequired(uniquePodName) // Remove any stored errors for the pod, everything went well in this processPodVolumes dswp.desiredStateOfWorld.PopPodErrors(uniquePodName) <ins> } else if dswp.podHasBeenSeenOnce(uniquePodName) { // For the Pod which has been processed at least once, even though some volumes // may not have been reprocessed successfully this round, we still mark it as processed to avoid // processing it at a very high frequency. The pod will be reprocessed when volume manager calls // ReprocessPod() which is triggered by SyncPod. dswp.markPodProcessed(uniquePodName) </ins> } }

It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!


Question: "k8s.io/kubernetes/pkg/api/legacyscheme", "k8s.io/kubernetes/pkg/api/service", "k8s.io/kubernetes/pkg/api/v1/pod", <ins> "k8s.io/kubernetes/pkg/api/v1/resource", "k8s.io/kubernetes/pkg/api/v1/service", </ins> "k8s.io/kubernetes/pkg/apis/apps", "k8s.io/kubernetes/pkg/apis/apps/validation", "k8s.io/kubernetes/pkg/apis/autoscaling",

What happened: There is to manage import ways under but that doesn't work at our upstream CI. We can enable it by But if doing that, we will face the following error with What you expected to happen: I hope the check is always running to block invalid import ways at upstream CI system. How to reproduce it (as minimally and precisely as possible): As line into arguments.InputDirs of
/sig testing /area e2e-test-framework


Question: "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/client-go/util/retry" </ins> v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/util/rand" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait"

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: package fake import ( <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/runtime"

Opening this issue based on what I ran into on with namespace as follows While unit testing the controller, we create a fake client and setup the controller to run using the client. This ends up creating a having the namespace set to When the call eventually reaches The action gets associated with a namespace only if the object has a namespace associated with it. Which in this will be Which will result in the error in to manage the event like does. In case if the namespace is set to Above is a test case that can easily simulate the issue in question. Referring back to the test under PR NA <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: for i := range volumes { vol := &volumes[i] // so we can set default values el := errs.ErrorList{} <del> // TODO(thockin) enforce that a source is set once we deprecate the implied form. if vol.Source != nil { el = validateSource(vol.Source).Prefix("source") </del> <ins> if vol.Source == nil { // TODO: Enforce that a source is set once we deprecate the implied form. vol.Source = &api.VolumeSource{ EmptyDir: &api.EmptyDir{}, } </ins> } <ins> el = validateSource(vol.Source).Prefix("source") </ins> if len(vol.Name) == 0 { el = append(el, errs.NewFieldRequired("name", vol.Name)) } else if !util.IsDNSLabel(vol.Name) {

Given the following configuration: No volume will be mounted inside the resulting container: If I modify the volumes element to look like this: I get a volume mounted on /srv: I see the same behavior using either or .
I suspect I broke this default behavior. Hopefully someone else can provide a fix? I'll fix if it's still pending in a day or two. (In the middle of other things)
I will look into this.


Question: // Initialize passes a Kubernetes clientBuilder interface to the cloud provider func (vs *VSphere) Initialize(clientBuilder controller.ControllerClientBuilder) { <ins> } // Initialize Node Informers func (vs *VSphere) SetInformers(informerFactory informers.SharedInformerFactory) { </ins> if vs.cfg == nil { return } // Only on controller node it is required to register listeners. // Register callbacks for node updates <del> client := clientBuilder.ClientOrDie("vsphere-cloud-provider") factory := informers.NewSharedInformerFactory(client, 5*time.Minute) nodeInformer := factory.Core().V1().Nodes() nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ </del> <ins> glog.V(4).Infof("Setting up node informers for vSphere Cloud Provider") nodeInformer := informerFactory.Core().V1().Nodes().Informer() nodeInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ </ins> AddFunc:    vs.NodeAdded, DeleteFunc: vs.NodeDeleted, }) <del> go nodeInformer.Informer().Run(wait.NeverStop) glog.V(4).Infof("vSphere cloud provider initialized") </del> <ins> glog.V(4).Infof("Node informers in vSphere cloud provider initialized") </ins> } // Creates new worker node interface and returns

In 1.9 vSphere cloud provider creates a new client, new shared informer factory and lists nodes IMO, the cloud provider should not create it's own client at all. GCE uses , called by controller-manager to initialize the cloud provider. This way, GCE does not need extra policy and it shares the informer with all the controllers. is called only in controller-manager process, i.e. for provisioning and attach/detach. Do you need the informer really in kubelet and/or API server? It's quite odd to list all nodes there. /kind bug /sig storage /sig auth
issue regarding is getting fixed with creating and binding with ServiceAccount. See is there a way in kubernetes, we can fix this without assigning permissions to serviceaccount? We will look into SetInformer interface GCE uses.
cloud provider specific roles should be applied externally as we prepare for moving cloud provider specific components out of tree. had feedback on the way that was wired in that would be good to adjust before copying that pattern
thanks for the pointer to old issue. I reworked this one to remove vsphere's own client (and policy).
/assign rohitjogvmw
GitHub didn't allow me to assign the following users: rohitjogvmw. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign rohitjogvmw Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Running into this same issue (manifested as "Kubernetes node nodeVmDetail details is empty" error upon pvc build using vsphere). Commenting to watch progress. Happy to help debug if needed.
/assign divyenpatel

Here if we with , then we are only able to watch the latest change. In this case if the key is modified after the returns but before the we call the , then we will miss the event. To avoid the window time, we'd better specify the to be the index in the response of the . BTW, I think if we use with , we can probably eliminate the first call.
This is fixed in


Question: // Initialize passes a Kubernetes clientBuilder interface to the cloud provider func (vs *VSphere) Initialize(clientBuilder controller.ControllerClientBuilder) { <ins> } // Initialize Node Informers func (vs *VSphere) SetInformers(informerFactory informers.SharedInformerFactory) { </ins> if vs.cfg == nil { return } // Only on controller node it is required to register listeners. // Register callbacks for node updates <del> client := clientBuilder.ClientOrDie("vsphere-cloud-provider") factory := informers.NewSharedInformerFactory(client, 5*time.Minute) nodeInformer := factory.Core().V1().Nodes() nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ </del> <ins> glog.V(4).Infof("Setting up node informers for vSphere Cloud Provider") nodeInformer := informerFactory.Core().V1().Nodes().Informer() nodeInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ </ins> AddFunc:    vs.NodeAdded, DeleteFunc: vs.NodeDeleted, }) <del> go nodeInformer.Informer().Run(wait.NeverStop) glog.V(4).Infof("vSphere cloud provider initialized") </del> <ins> glog.V(4).Infof("Node informers in vSphere cloud provider initialized") </ins> } // Creates new worker node interface and returns

Here if we with , then we are only able to watch the latest change. In this case if the key is modified after the returns but before the we call the , then we will miss the event. To avoid the window time, we'd better specify the to be the index in the response of the . BTW, I think if we use with , we can probably eliminate the first call.
This is fixed in


Question: Ports:     []api.EndpointPort{{Name: "", Port: 80}, {Name: "p", Port: 93}}, }}, }, <ins> { ObjectMeta: metav1.ObjectMeta{ Name:      "foo-second-ip", Namespace: metav1.NamespaceDefault, }, Subsets: []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{IP: "2001:db7::", TargetRef: &api.ObjectReference{Name: "foo", Namespace: metav1.NamespaceDefault}}}, Ports:     []api.EndpointPort{{Name: "", Port: 80}, {Name: "p", Port: 93}}, }}, }, </ins> }, } pods := &api.PodList{

Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


Question: }() } <ins> // StartRecordingToSink starts sending events received from the specified eventBroadcaster to the given sink. func (e *eventBroadcasterImpl) StartRecordingToSink(stopCh <-chan struct{}) { go wait.Until(e.refreshExistingEventSeries, refreshTime, stopCh) go wait.Until(e.finishSeries, finishTime, stopCh) e.startRecordingEvents(stopCh) } </ins> type eventBroadcasterAdapterImpl struct { coreClient          typedv1core.EventsGetter coreBroadcaster     record.EventBroadcaster

Which jobs are flaking: periodic-kubernetes-bazel-test-master (xref ) Which test(s) are flaking: TestEventSeriesf Testgrid link: Reason for failure: Timeout: Anything else we need to know: To reproduce with stress: /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
filed to address this issue.
contains the correct analysis of flakes (thanks a lot!), I commented on way of fixing it (which I'm not convinced about).


Question: args: - /nginx-ingress-controller - --default-backend-service=kube-system/default-http-backend <ins> - --publish-service=$(POD_NAMESPACE)/nginx-ingress-lb </ins>

Hello sig-network, We, sig-scalability, are exploring using "private" clusters (i.e. the ones where nodes don't have public IPs) for our large (up to 5K nodes) scalability tests to avoid consuming expensive public IPs. More details in With a few minor modifications we managed to get an almost passing correctness test suite on a private cluster except the 3 sig-network tests which explicitly assume that nodes are reachable over public IPs. Failing tests (): [sig-network] Firewall rule should have correct firewall rules for e2e cluster [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should be able to create a functioning NodePort service Could you take a look at them and advise if it would be possible to rewrite them so they can work both in the "public" and "private" clusters? We'd really appreciate your help assistance here. Let me know if you have any questions. Thanks, Matt
please triage /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
could you take a look?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
After the conversation I had with I'm un-assigning to make it go through the regular triage process. /unassign It's not a super high priority, but would be great if someone from the sig-network team could take a look. This effort could potentially result in releasing over 10K of public IPs.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
want to track this one with the release team as well as it is showing up in /priority critical-urgent /milestone v1.16
could we get someone to look into these issues or get an update of when we might get some news?
Reiterating the mentions to trigger a notification: <detailsIn response to : could we get someone to look into these issues or get an update of when we might get some news?

Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
should we add a new TAG on tests that does not apply to private clusters?
would adding tags to the tests skip private clusters entirely? I would think ideally private clusters should still have tests... if that's correct who would be responsible for adding more tests?
Hey Folks, sorry I must have missed updates here. I don't think that this should block 1.16 in any way nor it should be marked as priority/critical-urgent. This is an experimental test suite that we run in order to stop using public ips in scalability tests. What's more only 3 out of over 800 tests fail for private clusters and I'm quite confident that these tests don't fail because of a bug in kubernetes, but rather because they are written in a way that assumes that each node will have a public IP. The only action item here is to rewrite the tests and make them compatible with private clusters. I opened the bug to get some insights from sig/network on that as all failing tests belong to them. Removing the milestone and changing the priority, let me know if someone doesn't agree. /priority important-soon /milestone clear
/remove-priority critical-urgent
Thank you for the updates. I understand the test suite is experimental and there is a high level of confidence that the handful of failing sig-network jobs are failing due to the test suite and not to Kubernetes itself but given that this affects the quality of the signal we get from CI signal for this release we want to maintain it in the 1.16 milestone. could you bring this up with (don't see a group for test failures)? /milestone v1.16
friendly ping about this, would you be able to assign the correct person to write tests for this?
Let's bring this up in the sig-network meeting to see if we have any volunteers
Any updates on this issue?


Question: Ready:    true Hostname:   <unset> TargetRef:  Pod/test-124 <del> NodeName:   <unset> </del> <ins> NodeName:   node-2 </ins> Zone:       <unset> Events:         <none>` + "n", },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/uuid" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo"

Once the API for is finalized, add a node e2e test. /kind feature /assign /cc /sig node
We plan on adding this for 1.10


Question: echo "Creating minions." <ins> # TODO(mbforbes): Refactor setting scope flags. </ins> local -a scope_flags=() if (( "${#MINION_SCOPES[@]}" > 0 )); then scope_flags=("--scopes" "${MINION_SCOPES[@]}")

See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.


Question: klog.ErrorS(err, "inotify watcher error") } close(done) <ins> watcher.Close() </ins> }() return nil }

kubelet increased more than 100MB in memory in 4 days. kubelet to not used that amount of RAM I have reproduced it using 3 cronjobs which create hello world pods each minute and another cronjob that writes a line to a file each minute. Kubernetes version (use ): v1.18.1 Cloud provider or hardware configuration happens on both virtualbox and baremetal OS (e.g: ): centos 7 Kernel (e.g. ): 18.0-147.3.1 I have seen some existing issues like but I do not have any dangling resources associated with the cgroup.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign This is a duplicate issue, there has been a fixed PR but it has not been merged.

What happened: i wrote a shell then the kubelet's RSS increases continuously and does not release i make some trace and found some watched objects have never been released even these objects have not existed in real word i notice the fucntion readEvent is producer for these abnormal objects and print the length of every eventsome of which is beyond the length of buf even when i increase the array by 10 timesthe problem is still there and the limits are broken through again and again. so i realized the process of reading cannot be blocked. now i a queue for the buf of every inotify event and another goroutine to consume the queue. everythings seems good for now. What you expected to happen: fixed the bug How to reproduce it (as minimally and precisely as possible): running the shell : Anything else we need to know?: Environment: Kubernetes version (use ):1.17.9 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /area kubelet
/triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I didn't repeat the question. Is there a way of stable recurrence? Thx!
you could use the shell i'v paste above, or make "add" more faster than "delete".....
/assign
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The version mentioned is quite old. Can you please try to repro this on supported version of Kubernetes? /unassign /triage needs-information /remove-triage unresolved
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: testsuites.CapBlock:               true, testsuites.CapPVCDataSource:       true, testsuites.CapControllerExpansion: true, <ins> testsuites.CapSingleNodeVolume:    true, </ins> } return initHostPathCSIDriver("csi-hostpath", capabilities,

What happened: When we run csi hostpath driver e2e tests on a multi-node cluster some tests failed because they assume that volumes are node independent but this is not the case for some drivers like hostpath. Related PR; What you expected to happen: Skip the tests that assume volumes are node independent for the drivers that don't support node independent volumes. /sig storage /assign


Question: ExpectedError: "does not contain declared merge key", }, }, <ins> { Description: "$deleteFromPrimitiveList of nonexistent item in primitive list should not add the item to the list", StrategicMergePatchRawTestCaseData: StrategicMergePatchRawTestCaseData{ Original: []byte(` mergingIntList: - 1 - 2 `), TwoWay: []byte(` $deleteFromPrimitiveList/mergingIntList: - 3 `), Modified: []byte(` mergingIntList: - 1 - 2 `), }, }, { Description: "$deleteFromPrimitiveList on empty primitive list should not add the item to the list", StrategicMergePatchRawTestCaseData: StrategicMergePatchRawTestCaseData{ Original: []byte(` mergingIntList: `), TwoWay: []byte(` $deleteFromPrimitiveList/mergingIntList: - 3 `), Modified: []byte(` mergingIntList: `), }, }, { Description: "$deleteFromPrimitiveList on nonexistent primitive list should not add the primitive list", StrategicMergePatchRawTestCaseData: StrategicMergePatchRawTestCaseData{ Original: []byte(` foo: - bar `), TwoWay: []byte(` $deleteFromPrimitiveList/mergingIntList: - 3 `), Modified: []byte(` foo: - bar `), }, }, </ins> } func TestCustomStrategicMergePatch(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Patching an object without finalizers using results in the given finalizers being . is idempotent in case there are other list elements than the removed one. I expected to also be idempotent in case the list is already empty. an arbitrary object: an arbitrary finalizer to the object via strategic merge patch: the finalizer again by using the directive: the same patch again: Kubernetes version (use ):
/sig api-machinery
Strange. It looks like attempting to delete a nonexistent primitive actually adds it. Below is a slightly different take on your example, client-side only and without any finalizers to start with. It tries to delete which is nonexistent and it ends up being : I also tested with some previous versions of kubectl (back to 1.10.2 even) and saw the same thing, so it appears to have always been this way.
I just checked: the same thing happens when directly talking to the API server (so it's not a kubectl-only bug):
that's good to know. But I guess, this is not expected behavior, right?
I don't think it is expected behavior. It is surprising to me at least. I am curious to know what someone who has more experience with Strategic Merge Patch has to say though.
looks like this was reported in as well
/cc /triage accepted
The best workaround for now is probably to json merge patch the whole list with the resourceVersion set. I don't know much about the code nor the bug, but it really looks like a bug rather than a feature, so we should be able to fix it without causing too much surprise to other users. We generally have limited interest fixing smp but that seems worthy. Is somone interested in looking at it?
/assign


Question: message RollingUpdateDeployment { // The maximum number of pods that can be unavailable during the update. // Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). <del> // Absolute number is calculated from percentage by rounding up. </del> <ins> // Absolute number is calculated from percentage by rounding down. </ins> // This can not be 0 if MaxSurge is 0. // By default, a fixed value of 1 is used. // Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: labelSelector := "" fieldSelector := "" timeout := time.Duration(0) <ins> wait := true </ins> filenames := []string{} recursive := false

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: ioStreams, _, _, errBuf := genericclioptions.NewTestIOStreams() cmd := NewCmdDrain(tf, ioStreams) <ins> var recovered interface{} </ins> sawFatal := false fatalMsg := "" func() { defer func() { // Recover from the panic below. <del> _ = recover() </del> <ins> recovered = recover() </ins> // Restore cmdutil behavior cmdutil.DefaultBehaviorOnFatal() }()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: "storage.k8s.io/v1/csidrivers":                                     "hL6j/rwBV5w=", "storage.k8s.io/v1/csinodes":                                       "Pe62DkZtjuo=", "storage.k8s.io/v1/storageclasses":                                 "K+m6uJwbjGY=", <del> "storage.k8s.io/v1/csistoragecapacities":                           "xeVl+2Ly1kE=", "storage.k8s.io/v1/volumeattachments":                              "vQAqD28V4AY=", "storage.k8s.io/v1beta1/csistoragecapacities":                      "xeVl+2Ly1kE=", </del> <ins> "storage.k8s.io/v1/csistoragecapacities":                           "4as6MA/kOg0=", "storage.k8s.io/v1/volumeattachments":                              "tJx/ezt6UDU=", "storage.k8s.io/v1beta1/csistoragecapacities":                      "4as6MA/kOg0=", </ins> "apps/v1/controllerrevisions":                                      "85nkx63pcBU=", "apps/v1/daemonsets":                                               "dd7pWHUlMKQ=", "apps/v1/deployments":                                              "8aSe+NMegvE=",

Storage version defaults to the preferred version for a given group, but can be overridden when constructing API server storage. This is done here: Reasons the preferred version might be overridden: A given resource/kind does not yet exist in the preferred version (like when introducing a new alpha type in an existing API group with more stable versions) A given resource/kind is continuing to store in etcd in an older version for compatibility with n-1 API servers The etcd integration test verifies that we're actually storing the correct versions in etcd, but the computation of the StorageVersionHash is not honoring the overridden version. on master, this code reports v1 CSIStorageCapacity: even though the etcd test verifies we're still persisting v1beta1 /kind bug /priority important-soon /sig api-machinery cc Originally posted by in
/cc
cc
hi I have found that on points to but here the points to here is screenshot from debug console ! As I am still new to the codebase, I am trying to understand why it is happening so. But we can say that because of this reason, test case below does not failed because the expected hash we put in file matches correctly with the onc we create in at line 404 I also believe that the value corresponding to key has been wrongly put same as Correct values are as follows. One caveat here is, hashes are calculated with appending resource but the key for lookup is based on resource . I hope this information will help in locating the issue. Meanwhile, I will also try to look why in is pointing to even though we override.
/assign
The problem reported by this issue should not exist, on apiserver startup, we can get the right overrided version v1beta1 for the csistoragecapacities from , since the code below overrided the storage version ! But in the test , it did not override the storage version, see the code below so this test case reports v1 CSIStorageCapacity is right. do you think that we should change the testcase to override the storage version again to keep consistent and dispel the confusion for StorageVersionHash ?
Yes you are right. If we add below testcase will surely fail as the hash will not match in this case.

So, we need to update hash in to
ah, good catch... so this is a test-only issue, but still one we need to resolve, so the unit test reflects reality of storage versions when you actually start a kube-apiserver
As this pr would remove the storageversion override so after this pr merged, the testcase would be just right, then we don't need another pr to fix the testcase again, right?
But alteast we should update the correct hash value corresponds to in
the test would still be exercising preferred version ordering without any overrides, which would still be incorrect, and would resurface problems as soon as another override is
/triage accepted


Question: } // When a request is dequeued for service -> qs.virtualStart += G * width queue.nextDispatchR += request.totalWork() <del> qs.boundNextDispatchLocked(queue) request.decision.Set(decisionExecute) </del> return true }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned .

Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to .

hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .


Question: return waitForServiceAccountInNamespace(c, namespace, "default", ServiceAccountProvisionTimeout) } <ins> // WaitForKubeRootCAInNamespace waits for the configmap kube-root-ca.crt containing the service account // CA trust bundle to be provisioned in the specified namespace so that pods do not have to retry mounting // the config map (which creates noise that hides other issues in the Kubelet). func WaitForKubeRootCAInNamespace(c clientset.Interface, namespace string) error { return waitForConfigMapInNamespace(c, namespace, "kube-root-ca.crt", ServiceAccountProvisionTimeout) } </ins> // CreateTestingNS should be used by every test, note that we append a common prefix to the provided test name. // Please see NewFramework instead of using this directly. func CreateTestingNS(baseName string, c clientset.Interface, labels map[string]string) (*v1.Namespace, error) {

Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /cc
/hold I have a subtly different fix in :)
Oh wait, this is an issue. Yay, thanks :). Fixed in


Question: import ( "fmt" <ins> "net" </ins> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api"

Why doesn't the 'node' entity have a hostname and/or IP fields? Currently its id field (soon to become name field) holds the IP but is that something that a REST api consumer can firmly rely on to get the node's IP/hostname so this field 'name/id' will always hold IP/hostname and not other information? (IP/hostname is useful for example for visualization of the minions available in a k8s cluster in UI and showing their IP/hostnames)
Minion does have a HostIP field in v1beta1, and NodeStatus has a HostIP field in v1beta3. When you GET the resource, is the value for that field not set? If so, on what cloudprovider? It's true that there is no hostname field, but the minion/node name itself is the hostname.
, I'm currently using the hack/local cluster setup , this is the output of GET for minion 127.0.0.1 with v1beta1, seems that HostIP is not returned at all (even not with empty value) { "kind": "Node", "id": "127.0.0.1", "uid": "-8527-11e4-8f4b-", "creationTimestamp": "2014-12-16T15:31:50+02:00", "selfLink": "/api/v1beta1/minions/127.0.0.1", "resourceVersion": 7, "apiVersion": "v1beta1", "resources": { "capacity": { "cpu": 1000, "memory": } } } Can you please elaborate/point to docs about the change in v3? so HostIP field is moved into a NodeStatus section/internal to Node entity? BTW why is it considered part of a status? thank you
The best documentation of v1beta3 is unfortunately the source right now: It adheres to the conventions described here: The fact that HostIP is not reported is probably a quirk with static minions. In reality, hosts can have multiple IP addresses, some private and some public, v4 and v6, dynamically assigned (e.g., using DHCP), etc. For now, we support single, private, static IP addresses, but in cases where hosts are specified using hostnames, the IP addresses are looked up.

So, status makes the most sense to me.
Yeah, HostIP is only queried from cloudprovider. you won't see hostIP in json if it's empty (see the tag)
why HostIP is not returned for local static setup? the id reflects the IP anyhow. it sounds like a bug, but this issue is tagged with "documentation".
Yes, I agree that it's a bug that HostIP isn't populated. The documentation aspect is to explain somewhere why it's in Status rather than Spec.


Question: cleanupLocalVolume(config, testVol) }) <del> It("should be able to mount and read from the volume using one-command containers", func() { By("Creating a pod to read from the PV") readCmd := createReadCmd(volumeDir, testFile) podSpec := makeLocalPod(config, testVol, readCmd) //testFileContent was written during setupLocalVolume f.TestContainerOutput("pod reads PV", podSpec, 0, []string{testFileContent}) }) It("should be able to mount and write to the volume using one-command containers", func() { By("Creating a pod to write to the PV") writeCmd, readCmd := createWriteAndReadCmds(volumeDir, testFile, testVol.hostDir /*writeTestFileContent*/) writeThenReadCmd := fmt.Sprintf("%s;%s", writeCmd, readCmd) podSpec := makeLocalPod(config, testVol, writeThenReadCmd) f.TestContainerOutput("pod writes to PV", podSpec, 0, []string{testVol.hostDir}) }) </del> It("should be able to mount volume and read from pod1", func() { By("Creating pod1") pod1, pod1Err := createLocalPod(config, testVol)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign What happened: Looks like the event messaging has changed


Question: } // return: disk provisionState, diskID, error <del> func (c *ManagedDiskController) getDisk(diskName string) (string, string, error) { </del> <ins> func (c *ManagedDiskController) getDisk(resourceGroup, diskName string) (string, string, error) { </ins> ctx, cancel := getContextWithCancel() defer cancel() <del> result, err := c.common.cloud.DisksClient.Get(ctx, c.common.resourceGroup, diskName) </del> <ins> result, err := c.common.cloud.DisksClient.Get(ctx, resourceGroup, diskName) </ins> if err != nil { return "", "", err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: there will be azure disk creation issue when specifying an external resource group, after azure disk creation succeeded, it fails to get azure disk state What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /azure
/sig azure
/kind bug


Question: ContainersToKill:      getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, <del> "initialization in progress; do nothing": { </del> <ins> "an init container is stuck in the created state; restart it": { </ins> mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyAlways }, mutateStatusFn: func(pod *v1.Pod, status *kubecontainer.PodStatus) { status.ContainerStatuses[2].State = kubecontainer.ContainerStateCreated }, <del> actions: noAction, </del> <ins> actions: podActions{ SandboxID:             baseStatus.SandboxStatuses[0].Id, InitContainersToStart: []int{2}, ContainersToStart:     []int{}, ContainersToKill:      getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, </ins> }, "restartable init container has started; start the next": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyAlways },

It was consistently observed in a testing environment that a node-critical pod failed to start. event was present but was missing( is an init container). Also the following error was present but strangely only once: After some digging it seemed throwing an error matched the observations above. Eventually the init container and the pod to start The following tests in recreate the problem by and show how depending on the behavior is different. When the feature gate is enabled the last init container gets stuck in and never gets started. Therefore the pod get's stuck and never progresses The problem seems to stem from: There case get's skipped and the created container is expected to get started by the previous one by the following lines or But the provided tests show that this does not happen. Also if there is only one init container with the following logic should execute but won't because in this case this get's executed <details</details<details</details<details</details<details</details<details</details<details</details
/sig node can you help checking this issue report as according to it is a regression introduced with Thanks in advance!
I'll take a look. /assign
I guess this is a valid bug. /triage accepted /priority important-soon Can you provide a more specific scenario for this? Why do container runtime fail to start the container? (How can I reproduce it in e2e test?)
^ I guess this is the wrong assumption. If the container is in the state, the kubelet should resync it.
During node bootstrapping we had a bug where got restarted. So it is possible the following to happen: container gets created -restarts -container fails to start
We have a similar case. I have not yet looked at how containerd behaves at the moment, because it happens rarely and only at node startup. Pod stuck in the status. In the suppress status we have the following message: Alse we have the next events: Node characteristics: And it's like the error is reproduced on 1.29. The neighbouring cluster with k8s 1.28 is not affected by this problem
we have observed a similar behaviour but intermittently while upgrading our k8s cluster from k8s 1.29.0 to 1.29.6.

During upgrade process, when the node gets rebooted the "calico-node" pod of that node got stuck with below error forever until i restart the pod manually. Note: We are not observing this every time. Please let me know for any additional info required.
/cc


Question: hostPath: path: /dev containers: <del> - image: "gcr.io/gke-release/nvidia-gpu-device-plugin@sha256:a75ec0caa9e3038bd9886b3f36641a624574ff34b064974de6ee45048de3372b" </del> <ins> - image: "k8s.gcr.io/nvidia-gpu-device-plugin@sha256:4b036e8844920336fa48f36edeb7d4398f426d6a934ba022848deed2edbf09aa" </ins> command: ["/usr/bin/nvidia-gpu-device-plugin", "-logtostderr"] name: nvidia-gpu-device-plugin resources:

Looks like we have 2 images: Here: Should these be moved to community owned repositories? Are there already there and we just need to fix up the locations?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
for there are some PRs in progress to move the registry: and requested to copy the exiting image to the new community registry. for I will work on that next. :)
thanks !
discovered that the is available here: will open a PR to update to use this image
Where are the sources for these images? Will we be able to continue to publish new images? cc xref: None of these images seem to have version tags? If we're not actually publishing these, then we should probably either: consume them from somewhere that is publishing them and maintaining them set up mirroring them
copying from Also as much as I would like to see this sorted out I don't think or counts as a test-only issue to be resolving in this part of the release cycle. This seems like something we should do early in 1.22 and focus on actual test coverage improvements and functionality fixes right now.
/sig testing
/wg k8s-infra /milestone v1.22
Was CSI moved?
yes Ben ( was moved)
gcp-compute-persistent-disk-csi-driver is moved and nvidia-gpu-device-plugin have a good slack and the image was move in k/k with this PR I think we can close this issue. will close, if I did wrong we can reopen /close


Question: response := []*ec2.SecurityGroup{} clientRule := fmt.Sprintf("%s=%s", NLBClientRuleDescription, lbName) healthRule := fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, lbName) <ins> alreadyAdded := sets.NewString() </ins> for i := range securityGroups { for j := range securityGroups[i].IpPermissions { for k := range securityGroups[i].IpPermissions[j].IpRanges { description := aws.StringValue(securityGroups[i].IpPermissions[j].IpRanges[k].Description) if description == clientRule || description == healthRule { <del> response = append(response, securityGroups[i]) </del> <ins> sgIDString := aws.StringValue(securityGroups[i].GroupId) if !alreadyAdded.Has(sgIDString) { response = append(response, securityGroups[i]) alreadyAdded.Insert(sgIDString) } </ins> } } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After a rolling update of the cluster, NLB kept failing health checks because security group did not have rules for new EC2 instance. What you expected to happen: Security group should have been updated with both TCP rules for NLBs to work correctly. How to reproduce it (as minimally and precisely as possible): Create a service object with type LoadBalancer and adding the annotation for NLBs. FWIW, I had 6 services and each had their own NLBs. It’s possible that the bug happens when there are multiple NLBs causing multiple AWS updates. Perform a rolling update of the instance group. This causes the instances to be terminated and new ones to be brought up. AFAICT, this should result in two changes to the security group per EC2 instance. When the instance is terminated, the rules for that nodePort should be deleted. When the new node is created, the rule should be again. However, after the rolling update, the rule doesn’t get . As a result, the NLB’s health checks fail. Anything else we need to know?: As mentioned earlier, this happened on a cluster where there were multiple NLBs. This doesn’t seem to be a deterministic failure. The log has the following error message:I manually edited the Service object to see if it would simply retrigger an update on the cloud resources. It actually did. The rules got to the security group. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kops 1.8.1 Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you find any workarounds? we've run into this issue, where the security group rules got deleted. Specifically, we found that the security group missed the following: 1) TCP rule to allow traffic from NLB to Node Port 2) TCP rule to allow traffic from NLB to Healthcheck Port We these manually back in to fix, but were surprised these were deleted in the first place.

This stackoverflow also is relevant:
One workaround was to simply refresh the Kubernetes service object. We /updated a label in the Kubernetes service object and found that it would make the corresponding changes in AWS.
thanks have you been able to root cause why the rule doesn't get ? We're trying to re-produce the failure ourselves, too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Any update on this bug fix release please?
Sorry to bother you with commenting on this issue. But I cannot find out in what version of Kubernetes this is expected to be released / was released. Any clarification highly appreciated.