Question: } jig := e2eservice.NewTestJig(cs, ns, serviceName) <ins> jig.ExternalIPs = true </ins> ginkgo.By("creating service " + serviceName + " with type=clusterIP in namespace " + ns) clusterIPService, err := jig.CreateTCPService(func(svc *v1.Service) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: "k8s.io/kubernetes/pkg/api/v1" extensions "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" "k8s.io/kubernetes/test/e2e/framework" <del> "k8s.io/kubernetes/test/e2e/generated" </del> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


Question: package core import ( <ins> "fmt" </ins> "testing" "time"

If a Pod is subjected to a with "PriortyClass" as its scope, the quota's calculation is incorrect. See repo steps for more details. with "PriortyClass" as its scope can be calculated at all times. Create a PriorityClass named "p1"Create a quota with as its scope. Note: w/o specifying , this issue won't happenCreate a Job and wait for its completion.Check quota's usage calculation Here is incorrect - it should be . Worse calculation may happen if I create a regular pod and then delete the Job:The eventual result is totally incorrect. No response master, and possibly all versions. kind <details</details<details</details<details</details<details</details
/sig api-machinery /kind bug
/triage accepted
I would like to take this up!
is already in flight for this


Question: } w.resultsManager.Set(w.containerID, result, w.pod) <del> ProberResults.With(w.proberResultsMetricLabels).Set(result.ToPrometheusType()) </del> if w.probeType == liveness && result == results.Failure { // The container fails a liveness check, it will need to be restarted.

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: } } } <ins> func TestGetSelfLink(t *testing.T) { var testCases = []struct { desc             string name             string namespace        string expectedSelfLink string }{ { desc:             "No namespace specified", name:             "foo", namespace:        "", expectedSelfLink: "/api/v1/namespaces/default/pods/foo", }, { desc:             "Namespace specified", name:             "foo", namespace:        "bar", expectedSelfLink: "/api/v1/namespaces/bar/pods/foo", }, } for _, testCase := range testCases { selfLink := getSelfLink(testCase.name, testCase.namespace) if testCase.expectedSelfLink != selfLink { t.Errorf("%s: getSelfLink error, expected: %s, got: %s", testCase.desc, testCase.expectedSelfLink, selfLink) } } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): selflink kubelet selflink default selflink manifest selflink Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): admin kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T10:00:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T09:42:05Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: AWS OS: Debian GNU/Linux 8 (jessie) Kernel: Linux ip-10-150-109-115 4.4.41-k8s SMP Mon Jan 9 15:34:39 UTC 2017 x86_64 GNU/Linux Install tools: kops 1.5.3 Others: What happened: When running the selfLink for pods started from manifests were incorrect. Example: What you expected to happen: To receive a correct selfLink: Example: How to reproduce it (as minimally and precisely as possible): I believe simply using a pod manifest to start a pod will cause this issue as kubelet seems to rely on defaults when starting pods from manifests. Anything else we need to know: I believe the below linked line needs to be corrected:
/assign I reproduced it. Will fix. Thanks for reporting!
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: // shutting it down. If the pod hasn't started yet, we know that when // the pod worker is invoked it will also avoid setting up the pod, so // we simply avoid doing any work. <del> if !kl.podWorkers.IsPodTerminationRequested(pod.UID) { </del> <ins> // We also do not try to admit the pod that is already in terminated state. if !kl.podWorkers.IsPodTerminationRequested(pod.UID) && !podutil.IsPodPhaseTerminal(pod.Status.Phase) { </ins> // We failed pods that we rejected, so activePods include all admitted // pods that are alive. activePods := kl.filterOutInactivePods(existingPods)

Pods that already succeeded and waiting for finalizer are being readmitted, fail to be admitted and moving from Succeeded Phase to Failed (admission failed) phase. The correct behavior would be to not try to re-admit the succeeded (or failed) Pod. Specific case: Node is exposing Devices using device plugin. Device plugin is implemented as a Pod Job's pod that is using the Device run to completion and waiting for Job Status finalizer kubelet is being restarted DevicePlugin pod is reconnecting to the kubelet. While it is reconnecting, kubelet don't have availability of devices kubelet re-admits the Pod, even though it is terminated already kubelet marks the pod as "failed to be admitted" But the same can happen with the variations of GC speed and other admission handlers. /sig node /priority important-soon /cc There is also a wider issue: , that concentrates on running pods. This issue is about terminated Pods, which is less controversial. <detailsFor sure on master, may be on earlier versions. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/triage accepted I think we have at least another issue mentioning the same behavior but observed for jobs, I'll look it up and crosslink later


Question: // GetPodStatus(pod *kubecontainer.Pod) so that Docker can avoid listing // all containers again. status, err := g.runtime.GetPodStatus(pod.ID, pod.Name, pod.Namespace) <del> if klog.V(6).Enabled() { klog.V(6).ErrorS(err, "PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name), "podStatus", status) </del> <ins> if err != nil { if klog.V(6).Enabled() { klog.V(6).ErrorS(err, "PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name), "podStatus", status) } else { klog.V(4).ErrorS(err, "PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name)) } </ins> } else { <del> klog.V(4).ErrorS(err, "PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name)) } if err == nil { </del> <ins> if klog.V(6).Enabled() { klog.V(6).InfoS("PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name), "podStatus", status) } else { klog.V(4).InfoS("PLEG: Write status", "pod", klog.KRef(pod.Namespace, pod.Name)) } </ins> // Preserve the pod IP across cache updates if the new IP is empty. // When a pod is torn down, kubelet may race with PLEG and retrieve // a pod status after network teardown, but the kubernetes API expects

I install kubernetes 1.22.2 calico 3.20.2 centos 8.4 kernel 4.18 when I create some pod,and sawed that kubelet status had some error my node and pod are all running Is this a bug of 1.22.2 Can any one help me to resolv this problem,thanks! kubelet had PLEG Write status error I installed the service in 3 hours No response No response No response /sig
/remove-kind failing-test /sig node /kind support
In this pr , the log level was changed from Info to Error, may cause some confusion. /cc
/triage accepted
Is there a problem with this log message? Or is there no problem with that?
Yeah, there's no problem
But when I delete all the pods, I got some error kubelet log: did you know Do you know what the problem is : Error getting node" err="node "" not found"
ignore Flag --network-plugin has been deprecatedbecause i use kubernetes 1.22.2 and docker 20.10.10Docker is no longer officially recommended
You should see the actual realtime logs by "journalctl -f -u kubelet.service"
I restart kubelet,and use command "journalctl -f -u kubelet.service" . the log "Error getting node" err="node "" not found" is gone ! when i check my docker.service ,I get this : ! it doesn't have any error about docker.
Is that still a problem ?
I think there are no real errors in there. I think we should only log them if there is an error. /assign
/remove-kind support /kind bug
/retitle Kubelet logging PLEG: Write status error when no errors have occured


Question: return } <ins> // truncatePodHostnameIfNeeded truncates the pod hostname if it's longer than 63 chars. func truncatePodHostnameIfNeeded(podName, hostname string) (string, error) { // Cap hostname at 63 chars (specification is 64bytes which is 63 chars and the null terminating char). const hostnameMaxLen = 63 if len(hostname) <= hostnameMaxLen { return hostname, nil } truncated := hostname[:hostnameMaxLen] glog.Errorf("hostname for pod:%q was longer than %d. Truncated hostname to :%q", podName, hostnameMaxLen, truncated) // hostname should not end with '-' or '.' truncated = strings.TrimRight(truncated, "-.") if len(truncated) == 0 { // This should never happen. return "", fmt.Errorf("hostname for pod %q was invalid: %q", podName, hostname) } return truncated, nil } </ins> // GeneratePodHostNameAndDomain creates a hostname and domain name for a pod, // given that pod's spec and annotations or returns an error. func (kl *Kubelet) GeneratePodHostNameAndDomain(pod *api.Pod) (string, string, error) { // TODO(vmarmol): Handle better. <del> // Cap hostname at 63 chars (specification is 64bytes which is 63 chars and the null terminating char). </del> clusterDomain := kl.clusterDomain <del> const hostnameMaxLen = 63 </del> podAnnotations := pod.Annotations if podAnnotations == nil { podAnnotations = make(map[string]string)

Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here


Question: Persist(map[string]string) error } <ins> type noopPersister struct{} func (n *noopPersister) Persist(_ map[string]string) error { // no operation persister return nil } </ins> // All registered auth provider plugins. var pluginsLock sync.Mutex var plugins = make(map[string]Factory)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: no panicking How to reproduce it (as minimally and precisely as possible): use azure or OIDC auth plugin with a or without a Anything else we need to know?: a similar OIDC panic issue is registered here
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig auth

Using with a without a () will result in a panic. What happened: What you expected to happen: Expected a successful kube-api server call, or at the very least an error to be returned. Although i think it should just not call and continue with the kube-api server call. Anything else we need to know?: The reason for this is due to where has no check.
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> v1.31.0-go1.23rc2-bullseye.0 </del> <ins> v1.31.0-go1.22.5-bullseye.0 </ins>

sgtm, add an issue labeled as release-blocking and in the milestone to either promote to 1.23.0 or revert this to 1.22.x before our 1.31 release Originally posted by in Kubernetes is being built and tested with go1.23rc2 at HEAD () to let it bake in the CI and help iron out issues, if any, with this version of Go. However, for the 1.31 release, the version of Go should be a non-rc, released version. The release timeline of go1.23.0 is ~same as the release timeline of K8s 1.31.0 (see ). If go1.23.0 is not out by then, we should revert to go1.22.x before cutting K8s 1.31.0. /sig release architecture /area code-organization /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /cc
/milestone v1.31
cc
xposting relevant slack thread:
Also unlike in the past, missing shipping 1.23.0 in 1.31.0 is not a big deal, we have much more room to switch to 1.23.0 in a patch release after all the compat changes with the go team. So we should not fear reverting if the timelines don't line up.
(though go1.23 adds the thing that makes it workable to default to old go version compat settings, so we'll have to eke out another 5 months on 1.21/1.22 for until we can adopt 1.23 on release branches in ~Jan if we miss 1.31, but that's how it looks like it will have to happen)


Question: # DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips tallclair </del> <ins> pwittrock seans3 soltysh </ins>

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-cli-plugin repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig cli

The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the cli-runtime repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig cli


Question: <ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. TAG ?= e2e PREFIX ?= gcr.io/k8s-testimages all: push image: docker build --pull -t $(PREFIX)/redis:$(TAG) . push: image gcloud docker -- push $(PREFIX)/redis:$(TAG) clean: </ins>

Kubernetes version (use ): master What happened: Pulling the redis image takes a while, like 30 seconds in GCE. This might be leading to periodic timeouts in tests like . What you expected to happen: Tests shouldn't depend on large images that take a potentially long time to pull. proposed fix to reduce flakiness we can increase the replica count on this test to 2 or make the redis image we use smaller (or not use redis at all and just use a simpler smaller image)
cc (considering this as a bug since we shouldnt need a 500MG image just to test kubectl :))
This should be as simple as changing the current to use alpine and install redis, instead of basing it off debian, installing gcc, and compiling redis from source :-) The image should be ~10MB.


Question: }, } for k, v := range errorCases { <del> if errs := validateContainers(v, volumeDevices, field.NewPath("field")); len(errs) == 0 { </del> <ins> if errs := validateContainers(v, false, volumeDevices, field.NewPath("field")); len(errs) == 0 { t.Errorf("expected failure for %s", k) } } } func TestValidateInitContainers(t *testing.T) { volumeDevices := make(map[string]core.VolumeSource) capabilities.SetForTests(capabilities.Capabilities{ AllowPrivileged: true, }) successCase := []core.Container{ { Name:  "container-1-same-host-port-different-protocol", Image: "image", Ports: []core.ContainerPort{ {ContainerPort: 80, HostPort: 80, Protocol: "TCP"}, {ContainerPort: 80, HostPort: 80, Protocol: "UDP"}, }, ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, { Name:  "container-2-same-host-port-different-protocol", Image: "image", Ports: []core.ContainerPort{ {ContainerPort: 80, HostPort: 80, Protocol: "TCP"}, {ContainerPort: 80, HostPort: 80, Protocol: "UDP"}, }, ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, } if errs := validateContainers(successCase, true, volumeDevices, field.NewPath("field")); len(errs) != 0 { t.Errorf("expected success: %v", errs) } capabilities.SetForTests(capabilities.Capabilities{ AllowPrivileged: false, }) errorCases := map[string][]core.Container{ "duplicate ports": { { Name:  "abc", Image: "image", Ports: []core.ContainerPort{ { ContainerPort: 8080, HostPort: 8080, Protocol: "TCP", }, { ContainerPort: 8080, HostPort: 8080, Protocol: "TCP", }, }, ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, } for k, v := range errorCases { if errs := validateContainers(v, true, volumeDevices, field.NewPath("field")); len(errs) == 0 { </ins> t.Errorf("expected failure for %s", k) } }

I found that kubelet will not allocate hostports specified from initContainers. And PodFitsHostPorts in kube-scheduler does not take initContainer into consideration. Also, kube-apiserver think such pod is illegal, but I think it OK. We should make the behavior of spec.HostPorts in initContainer clear
/sig node
Exactly. I think this makes sense. Since initContainers are expected to be completed, we should not expose extra port mappings during creating pod sandbox, which may bring in security risks.
Sounds good to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: ) cidrSize := netutils.RangeSize(cidr) <del> if cidrSize < min { </del> <ins> // available addresses are always less than the cidr size // A /28 CIDR returns 16 addresses, but 2 of them, the network // and broadcast addresses are not available. if cidrSize <= min { </ins> return 0 }

kube-apiserver observed a panic: kube-apiserver should not panic Not sure yet -- will add more details as I investigate. No response v1.25.4 N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /cc
max is greater than offset?
/assign


Question: // Logout calls SessionManager.Logout for the given connection. func (connection *VSphereConnection) Logout(ctx context.Context) { <del> m := session.NewManager(connection.Client) </del> <ins> clientLock.Lock() c := connection.Client clientLock.Unlock() if c == nil { return } m := session.NewManager(c) </ins> hasActiveSession, err := m.SessionIsActive(ctx) if err != nil {

What happened: Data race in tests in an unrelated PR Logs https://k8s- /kind bug /area platform/vsphere /area cloudprovider /sig cloud-provider
/kind flake
Another time it flaked in an unrelated PR https://k8s-
bringing this to your attention as a vsphere dev.
/assign
Looks like the race was in Feb 2017 here: But that code path wasn't covered by go test until vcsim support was here: I'll work on a fix.


Question: . "github.com/onsi/gomega" ) <del> func isRestartNeverMirrorPod(p *api.Pod) bool { </del> <ins> func isNotRestartAlwaysMirrorPod(p *api.Pod) bool { </ins> if !kubepod.IsMirrorPod(p) { return false } <del> return p.Spec.RestartPolicy == api.RestartPolicyNever </del> <ins> return p.Spec.RestartPolicy != api.RestartPolicyAlways </ins> } func filterIrrelevantPods(pods []*api.Pod) []*api.Pod { var results []*api.Pod for _, p := range pods { <del> if isRestartNeverMirrorPod(p) { </del> <ins> if isNotRestartAlwaysMirrorPod(p) { </ins> // Mirror pods with restart policy == Never will not get // recreated if they are deleted after the pods have // terminated. For now, we discount such pods.

The restart test is broke by The original issue is The reason is We fixed this by filtering out mirror pod in the test. However changed to which broke the workaround. A quick fix is to filter out non-RestartAlways pods. Because either or pods could succeed, and we can not deal with terminated mirror pods very well now. /cc
Yeah... I see there's a number of problems here. I looked at the history of the image puller config and it looked to me that it was 'Never' from the beginning... Thanks for the fix though.
is this a release blocker for 1.5? Please update the issue ASAP, thanks! cc:


Question: args = append(args, "-j", "DNAT", "--to-destination", net.JoinHostPort(proxyIP.String(), strconv.Itoa(proxyPort))) return args } <ins> func isTooManyFDsError(err error) bool { return strings.Contains(err.Error(), "too many open files") } </ins>

Restarting might also be acceptable. It's a pretty painful user experience right now if this happens-- it fills the logs with this message.
Yes, it should just panic() , Daniel Smith wrote:
Yeah I guess if it's out of FDs it can't really open a network connection to file an event...
kube-proxy should send an event when it starts up, just like kubelet does with its method. This works around the fact that you can't send an event when you are out of FDs, and it still provides a lot of debugging value.
Agreed with 1) kube-proxy should panic when hitting out of fd 2) restarted kube-proxy should pick up the crash reason like what we do for internal agent 3) crash reason should be propogated to upstream as an event.
Moving to the right sig.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale
/area kube-proxy
we're going to close this because, after 3 years or so , decreasing interest in the userspace proxy and nobody other then some openshift folks, are using the linux kube proxy .. can reopen if necessary ping :)
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: source /run/flannel/subnet.env source /etc/default/docker <del> echo DOCKER_OPTS="${DOCKER_OPTS} -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock  </del> <ins> echo DOCKER_OPTS=" -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock  </ins> --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}" > /etc/default/docker sudo service docker restart }

Kubernetes version (use ): 1.3.7 Environment: Cloud provider or hardware configuration: Bare-metal Ubuntu cluster OS (e.g. from /etc/os-release): Ubuntu 14.04.3 LTS Kernel (e.g. ): 3.19.0-47-generic Install tools: KUBERNETESPROVIDER=ubuntu ./kube- Others: Docker version 1.12.1 What happened: appends the same copy of each time we start and stop Kubernetes cluster (using Ubuntu's & ). For example, if we start Kubernetes cluster the first time and restart it again, the content of is doubled: This cause the following error when starting Kubernetes as seen in : What you expected to happen: content should not be concatenated every time Kubernetes is restarted. How to reproduce it (as minimally and precisely as possible): Start Kubernetes, stop it, and start it again, using and . Anything else do we need to know: One possible fix is to remove the in the following line in :
This is a bug and easy to fix. I'll open a PR to fix it
Bug has been fixed.


Question: } func autoConvert_v1alpha1_DebuggingConfiguration_To_config_DebuggingConfiguration(in *DebuggingConfiguration, out *config.DebuggingConfiguration, s conversion.Scope) error { <del> out.EnableProfiling = in.EnableProfiling out.EnableContentionProfiling = in.EnableContentionProfiling </del> <ins> if err := v1.Convert_Pointer_bool_To_bool(&in.EnableProfiling, &out.EnableProfiling, s); err != nil { return err } if err := v1.Convert_Pointer_bool_To_bool(&in.EnableContentionProfiling, &out.EnableContentionProfiling, s); err != nil { return err } </ins> return nil } func autoConvert_config_DebuggingConfiguration_To_v1alpha1_DebuggingConfiguration(in *config.DebuggingConfiguration, out *DebuggingConfiguration, s conversion.Scope) error { <del> out.EnableProfiling = in.EnableProfiling out.EnableContentionProfiling = in.EnableContentionProfiling </del> <ins> if err := v1.Convert_bool_To_Pointer_bool(&in.EnableProfiling, &out.EnableProfiling, s); err != nil { return err } if err := v1.Convert_bool_To_Pointer_bool(&in.EnableContentionProfiling, &out.EnableContentionProfiling, s); err != nil { return err } </ins> return nil }

What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /sig api-machinery /cc
The label(s) cannot be applied. These labels are supported: <detailsIn response to : What would you like to be : Change the external DebuggingConfiguration member variables to pointer types. Why is this needed: So that we can set their default values to something different from golang's, for example enable profiling by default. /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
done in


Question: } for _, tc := range testCases { <ins> expBindAddr := tc.bindAddress if tc.bindAddress[0] == '"' { // Surrounding double quotes will get stripped by the yaml parser. expBindAddr = expBindAddr[1 : len(tc.bindAddress)-1] } </ins> expected := &componentconfig.KubeProxyConfiguration{ <del> BindAddress: tc.bindAddress, </del> <ins> BindAddress: expBindAddr, </ins> ClientConnection: componentconfig.ClientConnectionConfiguration{ AcceptContentTypes: "abc", Burst:              100,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: } endpointslicemetrics.EndpointSliceSyncs.WithLabelValues(metricLabel).Inc() } <ins> func dropEndpointSlicesPendingDeletion(endpointSlices []*discovery.EndpointSlice) []*discovery.EndpointSlice { n := 0 for _, endpointSlice := range endpointSlices { if endpointSlice.DeletionTimestamp == nil { endpointSlices[n] = endpointSlice n++ } } return endpointSlices[:n] } </ins>

As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.


Question: # scp order matters scp -r $SSH_OPTS  saltbase/salt/generate-cert/make-ca-cert.sh  <ins> easy-rsa.tar.gz  </ins> "${KUBE_CONFIG_FILE}"  ubuntu/util.sh  ubuntu/minion/*

cluster/saltbase/salt/generate-cert/make-ca- sets but does not echo what it is doing, so in the case of an error there is no evidence of which command produced the error.


Question: return err } var err error <ins> hcRequestPath, hcPort := hc.RequestPath, hc.Port </ins> if hc, err = gce.ensureHttpHealthCheck(hc.Name, hc.RequestPath, int32(hc.Port)); err != nil || hc == nil { <del> return fmt.Errorf("Failed to ensure health check for %v port %d path %v: %v", name, hc.Port, hc.RequestPath, err) </del> <ins> return fmt.Errorf("Failed to ensure health check for %v port %d path %v: %v", name, hcPort, hcRequestPath, err) </ins> } hcLinks = append(hcLinks, hc.SelfLink) }

Because I can't get the fix submitted without an issue.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: drivers.InitISCSIDriver, drivers.InitRbdDriver, drivers.InitCephFSDriver, <del> drivers.InitHostpathDriver, drivers.InitHostpathSymlinkDriver, </del> <ins> drivers.InitHostPathDriver, drivers.InitHostPathSymlinkDriver, </ins> drivers.InitEmptydirDriver, drivers.InitCinderDriver, drivers.InitGceDriver,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: execScripts: []ExecArgs{ {"fsck", []string{"-a", "/dev/foo"}, "", nil}, {"lsblk", []string{"-nd", "-o", "FSTYPE", "/dev/foo"}, "", nil}, <del> {"mkfs.ext4", []string{"-E", "lazy_itable_init=0,lazy_journal_init=0", "-F", "/dev/foo"}, "", nil}, </del> <ins> {"mkfs.ext4", []string{"-F", "/dev/foo"}, "", nil}, </ins> }, expectedError: fmt.Errorf("Still cannot mount"), },

Here's logs from formatting a 10GB HDD disk in Azure as ext4: It took seven and a half minutes to complete . Here's me doing it manually from the node: And even from a container, somewhat emulating kubelet: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu (Xenial) 16.10 Kernel (e.g. ): Linux 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubernetes-anywhere / static pods with kubelet / kubelet in container What happened: It took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred What you expected to happen: It to take a matter of seconds, similar to how it does if I execute the command manully. How to reproduce it (as minimally and precisely as possible): Build with on top. Deploy a cluster. Try to attach an unformatted VHD. Observe along with watching logs and confirm it really does take that long. Anything else do we need to know: I'm running containerized. Like this:
Can you confirm you were not seeing this behavior during your testing? I assume you were not running containerized?
Even if I and execute the same command from there it executes in a matter of a few seconds.
I'm seeing this behavior even when running as a normal systemd unit, and not as a docker container under a unit. Here's : And I'm still seeing the very slow , so it seems not related to running kubelet in container. I have also tested to ensure that this isn't something some sort of host caching. Even on another blank disk being attached for the the first time, executes quickly except when run by .
I confirm this is this case on my setup running local-up- I have a 100GB HDD disk. It took 30 seconds to format on the console but 2 minutes by kubelet. Is there any I/O constraint imposed on kubelet?
to see if she knows why would be slow when called from the kubelet.
cc/ Who is working on node performance benchmark.

Can we quickly run an experiment on GCE to see if we can reproduce the issue. Let's first rule the impact of containerized Kubelet. If the issue is still exist, I suspect the issue in Kubelet mounting logic. We can enable some tracing, and find the bottleneck.
I tried formatting a 10GB HDD in GCE Kubernetes (not containerized). The time of formatting in kubelet log is ~1s. The log is:
On Azure, a 10GB disk took 18s to mkfs by kublet but 4s to mkfs on the console. Do you have a chance to test these two scenarios? Size probably is not a critical factor for gce since gce pd is provisioned at fixed IOPS per GB, while Azure (also AWS) disks are provisioned at fixed IOPS per volume.
There's more variables in play or else I'm seeing a different edge case than you are, On Azure with a cluster, a 10GB disk for me consistently takes 7+ minutes. (, standard storage, HDD type). It was happening with a and with premium storage and SSD type data disk too... I can make a cluster available to anyone who would like to investigate the exact behavior I've been observing. I can provide kubeconfig+ssh creds. Contact me on Slack (colemickens) if interested.
mkfs's kernel stack after started by kubelet is always as the following. Not sure if it related to blkio cgroup
I have actually created a branch of to give me a cluster where is never running in a container and I'm still seeing the exact same issue. Took 7.5 minutes again, based on kubelet logs and watching . I think we can safely eliminate "containerized kubelet" as being the source of the problem.
Though if I remember correctly, kubelet places itself in a cgroup when it starts up. ( can confirm) so it's possible that it is still due to cgroup I/O throttling.
Above test result proves my initial suspect that the root cause in 7-min use case is containerized kubelet. But recent experiment is not running in a container.

could you please provide us more information: 1) cat /proc/<kubelet-pid/cgroups to find kubelet's cgroup 2) check & blkio.weightdevice setting 3) ps ax | grep makfs to find process id, the find which cgroup the process belongs to 4) check 3) cgroup's and blkio.weight_device If both 2) and 4) don't have issues with cgroup setting, please check if there is parent cgroup for them, and their setting since both CFQ and throttling supports hierarchy.
FYI: Kubelet does not perform any block IO throttling as of now for itself or any other entity on the node. , Dawn Chen wrote:
Yes, we know Kubelet doesn't. But the issue here indicates containerized Kubelet's IOs is throttled, and we are trying to figure out why, especially the user is running kubernetes-anywhere on Azure, which we, node-team don't normally test.
Here's how I was running originally: Here's how I was running when testing it not in a container: I will try to collect the data you mention but I'm going to have to learn how to do some of it along the way, I don't know much about cgroups.
this is not limited to azure, saw it on aws as well.
it seems only the hit only happened to ext4, xfs didn't have this problem.
I ran across the same performance issues and tried to investigate. I use Azure Dynamic Disk provisioning to provision a 1TB volume and it takes very long time for mkfs to finish. Doing the "mkfs.ext4 -F -E lazyitableinit=0,lazyjournalinit=0 /dev/sdd" manually is also very slow, but I did not compare the performance to the call done from inside kubelet. While running the mkfs manually, I observed that iotop shows around 62mb/s and that the free memory immediately shrinks from 2000mb to 100mb. It looks like mkfs is using up all free memory for the buffer cache. I even observed oom situations in the kernel log and processes being killed because of this. I've seen this behavior while running mkfs manually and while it was run by kubelet.

Removing the extended flag "lazyitableinit=0" from the manual mkfs call fixes both problems, performance and memory/buffer cache usage. My solution will now be to use xfs (as rootfs reported it to work better). But IMHO the lazyitableinit should be completely removed from the mkfs call.
Can we try to make sure this gets addressed in 1.6?
I did some further investigation here and it turns out there are other factors involved as well, especially on Azure. My initial statement that I got OOM due to mkfs.ext4 seems to be not true, as the OOMs were caused by fluentd-elasticsearch. I tried to use mkfs.ext4 with "-E discard=1" as I assumed that it should speed up things dramatically. This did not make a difference however. While trying to understand the mke2fs code from e2fsprogs, I realized that discard is enabled by default and always performed as first step when the underlying block device supports this. From the mkfs.ext4 logs you can actually see that it is executed at the beginning. It turns out that mke2fs does only skip inode table initialization when the kernel reports that a discard also ensures zeroing of the data. mke2fs checks this by doing an ioctl to BLKDISCARDZEROES. Support can also be checked with On Azure however, the kernel reports that a discard does NOT zero out the discarded blocks. This results in a loop, using normal write calls to write a lot of zeros to the block dev. This completely fills up the page cache with dirty pages, slowing down the whole system extremely. If there is not much free memory available that can be used for write buffering, the write rate goes down to very low values as well. I did not check the kernel code for lazyitableinit, but I would assume that it uses much more efficient code to write the zeros to the block device (e.g. by not relying on dirty pages). I'd also assume that it does these checks for the discard+zeroout behavior/support and then use discard if possible. If my assumption is correct, then it should not affect performance in a bad way when lazy init would be enabled again. The kernel would then do the discards in a lazy fashion, but finish pretty fast.

On Azure, the lazy init would still take some time, but use much less system resources (page cache) as the non-lazy solution. And most importantly, the device can already be mounted after a few seconds. (UPDATE: please check next comment) I tried to figure out if a discard (using the blkdiscard cmd) does really leave you with non-zeroed data and it looks like it actually becomes zero. I can however not see if this happens immediately or with a delay. Also, in the Azure Portal I can see that the "Billable Size" goes from several GB back to 0 after a short time. My assumption is that on other cloud providers, e.g. GCE and AWS, the block devices report that discard zeroes out the data. Can anyone with a GCE/AWS machine check this with the cat call from above? This would explain why this slowness does not occur on non-Azure clouds. Can you check why the kernel reports that discard does not zero-out? Is this a bug in Azure maybe?
~Ok, my assumption about the kernel using discard for lazy init was wrong. Just checked the kernel source and it simply issues multiple write bios which write zero-pages. So it would definitely have a negative impact on other cloud providers when lazyitableinit would be enabled.~ Would it be a viable solution to check support for BLKDISCARDZEROES in kubernetes before doing the mkfs and then decide based on that if lazyitableinit should be enabled or not?
Argh, digging deeper into the kernel turned out that it actually uses discard for zeroing out (check and ). So my initial assumption was correct. Sorry for all the go home now :)
Just checked the discard support on AWS and it is not supported at all, so it always does a normal write to clear/init the inode tables.
here's the output from a GCP instance: We haven't seen any slowness on GCP, even though the above settings are identical to our Azure instance's. We're trying to solve the same slowness issue on Azure, just not with kubernetes :) (we're Cloud Foundry). We're not using containers on this particular VM. Also, the problem manifests with but not .

Related issue here:
We discovered that by using instead of the problem went away — disks which previously took 8+ minutes to mkfs.ext4 took seconds. Alternatively, will also work as a partitioner (i.e. the subsequent will finish within seconds). Really. We're not sure why the tool we use to write one sector (i.e. the MBR) on a disk would make such a difference. HTH,
Did you discover this by manually running / on the nodes, or did you actually modify to use ?
This sounds like alignment issues. As far as I remember, sfdisk does not do proper alignment and thus may result on very slow performance when you are not on classical spinning disks. Azure disk provisioner does not even create partitions on the disks and thus alignment should not be the problem here. Just tested creating a partition with sfdisk and iotop showed only 7-10mb/s throughput when doing mkfs. Not using a partition at all or using fdisk gives around 60mb/s.
: We discovered this by manually running gparted / sfdisk on the instances we deployed on Azure (i.e. we're not using containers). We don't use Kubernetes — we are developers on Cloud Foundry, which is an alternative to Kubernetes. When we ran into this problem, we were pleased to discover we were not the only ones who have been bitten by this bug. And we wanted to make sure our Kubernetes kin were aware of the fix that we found to be effective. some notes on our research (and I cannot thank you enough for your detailed descriptions): we also found throughput of 7-10MB/s when was healthy when things were unhealthy, showed throughput of ~300kB/s during . we found that by passing (use direct IO and bypass buffer cache), we doubled our performance to ~700kB/s, but it was still ridiculously long (3 minutes).
I am going to try to do some additional investigation tonight, but I'm unsure of how to translate your observations to Kubernetes here. As implied, we don't use anywhere here (that I'm aware of). There's three scenarios: -dynamically provisioned. User brings a pre-formatted VHD. Everything goes smoothly as the merely needs to mount the block device and tell to mount it into the container. -dynamically provisioned.

User attaches an empty disk to an Azure VM, detaches it, uses it in a normal AzureDisk volume on a Pod. This is the scenario in the original post, and you can see that it is being mkfs'd without any partitions. disk provisioning. The VHD footer is written into the page blob, and then formats the whole disk, unpartitioned (trusting description of the provisioner here). I'm hesitant to rule out an alignment issue since you're seeing similar symptoms as we are here, but I'm not sure how to reconcile that with my other points in this comment...
What do you think about It would solve this issue. It would not tell us why it is slower in kubelet, but the problem itself would be solved. Should I CC more people to get this discussed faster? I would even like to see this in an upcoming 1.5.x release.
a lead Azure Engineer ( ) found the problem — it is an alignment problem gone horribly wrong: In situation like unaligned partition boundary, a big I/O request (e.g. a 32KB write) can be split into 64 512-byte writes by the kernel block I/O subsystem in the worst case, easily hitting the 500-IOPS throttling of Azure. It was made to simplify Hyper-V storage driver’s code, so Hyper-V’s storage driver picked up this patch in kernel versions: 4.4+, that is why only Azure is impacted. The community member who submitted the original patch already for this issue, waiting for Linux community to merge.
Does that also affect cases where the disk is unpartitioned, ?
If I understand the description correctly, this only applies to cases where unaligned partitions are in use. This is not the case for Azure Dynamic Disk provisioning and also for all the tests performed so far by all the participants here. As noted in , I did some manual tests with partitioning. And having unaligned partitions really degrades performance a lot (reducing mkfs throughput from 60mb/s to 10mb/s). But as said, we don't have unaligned partitions when using Azure Dynamic Disk actually don't have any partitions at all as we create the FS directly on the block device (e.g. /dev/sdc instead of /dev/sdc1). The real issue here is that lazy inode table initialization is disabled for mkfs.ext4.

The reason for disabling it can be found here: The assumption (BLKDISCARD+BLKDISCARDZEROES support) made in this commit is not true for all cloud providers, e.g. Azure and at least for AWS magnetic disks. I did some calculations to figure out how much zeroes must be written by mkfs.ext4 to initialize the inode tables for a 1TB disk. If you check "tune2fs -l /dev/sdX" you'll see that there are inodes with a size of 256 bytes allocated for a 1TB file system. This gives a total of 16G! These 16G of zeroes are written by mkfs.ext4 when lazy init is disabled, in a very very naive way. Writing 16G of zeroes already takes 4:30 minutes when you have a throughput of 60mb/s. Now if there is something that throttles the troughput (and it looks like kubelet does this somehow), it gets even worse. And that's not all, while this is written do disk, the complete system suffers on low memory due to all of it being used for write caching/buffering. If other processes want to allocate memory in this situation, they're blocked until enough from the buffers is flushed to disk. Please check It fixes all this by simply re-enabling lazy inode table initialization. I also tried to describe why this change should not affect cloud providers that previously thought that disabling lazy init would be an improvement.
I don't believe the partition alignment issue would affect cases where the disk is unpartitioned (i.e. using the entire disk, e.g. ). I suspect that any use of an entire raw device would be properly aligned (any filesystem that starts on sector 0 of the hard disk would be properly aligned), but that's a hunch and nothing more. : I think your assessment is on target — the problem we (Cloud Foundry) are seeing is caused by a partition misalignment, but it seems that the problem described in this thread has a different cause. We currently use a lazy inode table initialization, so we're covered on that front:

According to : While the is exactly the opposite. This significantly slows down large ext4 volume creation.
This might be worth bringing up at the storage f2f coming up.I am going to add to the agenda. This is an interesting approach to what I think may be a few larger issues. Questions and thoughts: What is a good SLO for mkfs? If the SLO is going to be violated due to size should k8s do something or should the user pre-format? There are tradeoffs for initting the filesystem lazy or not. Supporting all the options for a user may require an API change. Is this a persistent volume only thing?
I may miss something, but on a random GCE node with 500GiB magnetic(!) disk and Fedora 24 takes around 2 seconds. ext3 is a different story, it takes 74 seconds.
I cannot imagine any reason why anyone would be installing ext2/ext3 on a 500GB drive. If they do they should be expecting things to be slow. In any case knowing what the SLO is is still not a crazy idea.
This is probably not a big problem with GCE, since GCE PD's IOPS is per GB. This is mostly a problem for EBS/Cinder/Azure, since the IOPS per volume is fixed.
If we're going to do this at all, we should really think hard about this problem top-to-bottom. Ext3 vs ext4 vs other FS? Myriad filesystem options, etc.
I got bitten by this when implementing e2e tests for all supported storage class options on AWS. "Cold HDD" volume type has max. 250 IOPS and the smallest one is 500 GB. It takes 3-4 minutes just to format the volume, which slows down the test quite considerably. Was there any conclusion on f2f meeting? was in commit , which claims to be rewritten to Go. Original script not exist any more on the link above. Was there some smart idea in original ?
see some inconclusive investigation at
Please take a look at I encountered not only performance issue due to lazyitableinit=0, but also oom situations and thus killed processes. Due to the amount of data written to the disk, all free memory gets filled with dirty buffer cache.


Question: false, "foo", }, <ins> { "two defaults with same creation time, choose the one with smaller name", []*storagev1.StorageClass{defaultClass1, defaultClass2, classWithFalseDefault, classWithNoDefault, classWithEmptyDefault}, claimWithNoClass, false, defaultClass1.Name, }, { "two defaults, choose the one with newer creation time", []*storagev1.StorageClass{classWithCreateTime1, classWithCreateTime2, classWithFalseDefault, classWithNoDefault, classWithEmptyDefault}, claimWithNoClass, false, classWithCreateTime1.Name, }, </ins> } for _, test := range tests {

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: pathRecorderMux := mux.NewPathRecorderMux("kube-scheduler") healthz.InstallHandler(pathRecorderMux, checks...) installMetricHandler(pathRecorderMux, informers, isLeader) <ins> if utilfeature.DefaultFeatureGate.Enabled(features.ComponentSLIs) { slis.SLIMetricsWithReset{}.Install(pathRecorderMux) } </ins> if config.EnableProfiling { routes.Profiling{}.Install(pathRecorderMux) if config.EnableContentionProfiling {

As per, , wire up metrics/slis for KCM
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig instrumentation


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/validation/field" admissionapi "k8s.io/pod-security-admission/admission/api" "k8s.io/pod-security-admission/admission/api/validation"

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: // kubelet is running in the host's root mount namespace. type Mounter struct{} <del> var _ = Interface(&Mounter{}) </del> // Mount mounts source to target as fstype with given options. 'source' and 'fstype' must // be an emtpy string in case it's not required, e.g. for remount, or for auto filesystem // type, where kernel handles fs type for you. The mount 'options' is a list of options,

Compilation error in GetDeviceNameFromMount declared twice. and are duplicates and one of them need to be reverted. (build is disabled for linux, that's why both of them got in)
see , please
Looks like this can be closed now. right ?
Yes, closed by


Question: While($true) { $data = Get-InstanceMetadata -Key "service-accounts/default/scopes" if ($data) { <del> return ($data -match "auth/devstorage") </del> <ins> return ($data -match "auth/devstorage") -or ($data -match "auth/cloud-platform") </ins> } Start-Sleep -Seconds 1 }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If GCE Windows VMs are created with API scope the download GCS code will not insert the bearer token when fetching artifacts. This means that private builds of Kubernetes won't download. GCE Windows VMs should pull Kubernetes from GCS if cloud-platform scope is set on the VMs service account credentials. The cluster will not come up with Windows nodes. You can then check Cloud Logging or Serial port output for the impacted VMs and see something like, Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign me
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign jeremyje


Question: <ins> { "swaggerVersion": "1.2", "apiVersion": "", "basePath": "https://10.10.10.10:6443", "resourcePath": "/apis/extensions", "apis": [ { "path": "/apis/extensions", "description": "get information of a group", "operations": [ { "type": "void", "method": "GET", "summary": "get information of a group", "nickname": "getAPIGroup", "parameters": [], "produces": [ "application/json" ], "consumes": [ "application/json" ] } ] } ], "models": {} } </ins>  No newline at end of file

Loading returns Looking at the code is registered in the same way as which works fine. So not sure why is this broken.


Question: def := ing.Spec.Backend ns := ing.Namespace if def == nil { <del> // Ingresses that don't specify a default backend inherit the // default backend in the kube-system namespace. def = &networkingv1beta1.IngressBackend{ ServiceName: "default-http-backend", ServicePort: intstr.IntOrString{Type: intstr.Int, IntVal: 80}, } ns = metav1.NamespaceSystem </del> <ins> w.Write(LEVEL_0, "Default backend:t<default>n") } else { w.Write(LEVEL_0, "Default backend:t%sn", i.describeBackendV1beta1(ns, def)) </ins> } <del> w.Write(LEVEL_0, "Default backend:t%s (%s)n", backendStringer(def), i.describeBackendV1beta1(ns, def)) </del> if len(ing.Spec.TLS) != 0 { describeIngressTLSV1beta1(w, ing.Spec.TLS) }

I ran on a fresh minikube with no other services in it and saw this output. Which suggests there is a service in the cluster called (there isn't), and further suggests there is something wrong (there isn't) because there are no endpoints found. I expected to provide me with an accurate description of my Ingress object. This behaviour appears to be hardcoded into the kubectl describe command. Which contains a comment suggesting that this is default behaviour, but no logic to check if the service actually exists before attempting to default to it. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
See also do you know the history here?
not beyond what shows: Looks like an (incorrect) assumption in the describer that the only implementation would behave in this way
/assign
work in progress...
when you get the PR up please link this issue and in it and ping me directly on slack to get a fast review. I agree with what was stated here and in the other issue, that this seems like an unnecessary assumption that should've not been there, we probably should just leave it empty or per print as the value.


Question: By("restricting to a time range") time.Sleep(1500 * time.Millisecond) // ensure that startup logs on the node are seen as older than 1s <del> out = runKubectlOrDie("log", pod.Name, containerName, nsFlag, "--since=1s") recent := len(strings.Split(out, "n")) out = runKubectlOrDie("log", pod.Name, containerName, nsFlag, "--since=24h") older := len(strings.Split(out, "n")) Expect(recent).To(BeNumerically("<", older)) </del> <ins> recent_out := runKubectlOrDie("log", pod.Name, containerName, nsFlag, "--since=1s") recent := len(strings.Split(recent_out, "n")) older_out := runKubectlOrDie("log", pod.Name, containerName, nsFlag, "--since=24h") older := len(strings.Split(older_out, "n")) Expect(recent).To(BeNumerically("<", older), "expected recent(%v) to be less than older(%v)nrecent lines:n%vnolder lines:n%vn", recent, older, recent_out, older_out) </ins> }) }) })

I noticed this flake by looking at the submit queue. The test is simple, but when it fails, we have no good way to debug it. Below is the test in question. When the test fails, we should print the lines we got in each case.


Question: enable_manifest_url: '$(echo "${ENABLE_MANIFEST_URL:-}" | sed -e "s/'/''/g")' manifest_url: '$(echo "${MANIFEST_URL:-}" | sed -e "s/'/''/g")' manifest_url_header: '$(echo "${MANIFEST_URL_HEADER:-}" | sed -e "s/'/''/g")' <del> master_name: '$(echo "${KUBERNETES_MASTER_NAME:-}" | sed -e "s/'/''/g")' </del> <ins> master_name: '$(echo "${MASTER_NAME:-}" | sed -e "s/'/''/g")' </ins> num_nodes: $(echo "${NUM_NODES:-}" | sed -e "s/'/''/g") e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")' <del> initial_etcd_cluster: '$(echo "${KUBERNETES_MASTER_NAME:-}" | sed -e "s/'/''/g")' </del> EOF if [ -n "${KUBELET_PORT:-}" ]; then cat <<EOF >>/srv/salt-overlay/pillar/cluster-params.sls

test grid: Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it for debs: The corresponding RPM job is also faster now:


Question: function start_kubescheduler { SCHEDULER_LOG=${LOG_DIR}/kube-scheduler.log <del> ${CONTROLPLANE_SUDO} "${GO_OUT}/hyperkube" scheduler  </del> <ins> ${CONTROLPLANE_SUDO} "${GO_OUT}/hyperkube" kube-scheduler  </ins> --v="${LOG_LEVEL}"  --leader-elect=false  --kubeconfig "${CERT_DIR}"/scheduler.kubeconfig

Hyperkube supports a number of short aliases for commands, such as , , , , etc, rather than the longer , , , etc. There are comments indicating this is "unfortunate" and we still currently create these aliases in the hyperkube docker image: It'd be nice to simplify this, assuming we can find all uses of the short aliases and remove them. /kind cleanup Not sure which sig should own this, so tagging a few. /sig architecture cluster-lifecycle cli cc
interestingly, the command doesn't create the short aliases, though we also don't really use anywhere either.
I think we can just go ahead and start the deprecation cycle for this. WDYT? cc
cc as well
+1 to nuke it
/assign :wink: ok?
SGTM
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on it!
It seems like this issue was already solved in ?
I don't think the hyperkube code was changed in that PR, only the image?
Yes, thanks for your comment, can i try to fix it ?
I made a PR here: . Review/feedback/suggestions appreciated! :)
/assign /unassign
GitHub didn't allow me to assign the following users: Rand01ph.

Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig architecture cli /sig release Does anyone even still use hyperkube? Wouldn't they use kubeadm or k3s instead?
AFAIK, SAP (the company) still use hyperkube.
aks-engine deployer ( ) still uses hyperkube
Since this ticket is completed now. Can this todo be removed?
we are removing the whole thing :)


Question: // DelEntry is used to delete the specified entry from the set. func (runner *runner) DelEntry(entry string, set string) error { <del> entry = strings.Split(entry, " comment")[0] </del> if _, err := runner.exec.Command(IPSetCmd, "del", set, entry).CombinedOutput(); err != nil { return fmt.Errorf("error deleting entry %s: from set: %s, error: %v", entry, set, err) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area kube-proxy /area ipvs /assign I'll send a fixing pull request tomorrow. What happened: On centos 7.2 with ipset 6.29, kube-proxy in ipvs mode failed to create ipset sets. After some dig, it turns out to be problem of ipset on centos/rh. See I tried ipset 6.29 on ubuntu, it's OK. But when I tried ipset 6.29 on centos, I got errors: What you expected to happen: Since the problematic ipset versions have very large installation base, we could remove ipset commet, or remove ipset commet with problematic ipset version. The problematic ipset also says it support comment by --help. The problem could be detected by a test run of "ipset create foo hash:ip comment". How to reproduce it (as minimally and precisely as possible): Create some service in kubernetes, run kube-proxy with ipvs mode on Centos 7.x node. Anything else we need to know?: Environment: Kubernetes version (use ): commit (actually all commits after ) Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: ipset v6.29, protocol version: 6
could you test it before commit , that's ok?
, yes, I did. It's OK. But I think the we don't need to revert the iptables comments.
if ipset package not installed, will set module or match-set option in rule be weird and now seems not.


Question: srcs = [ "doc.go", "exec.go", <ins> "exec_mount_unsupported.go", </ins> "fake.go", "mount.go", "mount_unsupported.go",

/kind bug What happened: pull-kubernetes-cross complains: https://k8s-
/sig storage
For reference, root cause


Question: } func (f *fakeService) WaitForReady(ctx context.Context) error { <del> err := wait.PollWithContext(ctx, 1*time.Second, 200*time.Millisecond, func(ctx context.Context) (done bool, err error) { </del> <ins> err := wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, time.Second, false, func(ctx context.Context) (done bool, err error) { </ins> return f.Port() != nil, nil })

CI always has error when using to replace in function in file . No error occurred Use to replace in function in file . No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /area test
/triage accepted /assign /assign


Question: "github.com/spf13/cobra" "k8s.io/cli-runtime/pkg/genericclioptions" <del> cmdutil "k8s.io/kubectl/pkg/cmd/util" </del> <ins> cmdtesting "k8s.io/kubectl/pkg/cmd/testing" </ins> ) <del> func TestNewCmdVersionWithoutConfigFile(t *testing.T) { tf := cmdutil.NewFactory(&genericclioptions.ConfigFlags{}) </del> <ins> func TestNewCmdVersionClientVersion(t *testing.T) { tf := cmdtesting.NewTestFactory().WithNamespace("test") defer tf.Cleanup() </ins> streams, _, buf, _ := genericclioptions.NewTestIOStreams() o := NewOptions(streams) if err := o.Complete(tf, &cobra.Command{}); err != nil {

/sig cli run (or just ). observe failure: xref:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I have filed to address this issue.
Thank you!


Question: removeFn = append(removeFn, list.Enqueue(arrival[i])) } <del> dequeued := make([]*request, 0) for _, f := range removeFn { dequeued = append(dequeued, f()) </del> <ins> expected := append([]*request{}, arrival...) for idx, f := range removeFn { if a := f(); a != arrival[idx] { t.Errorf("Removal %d returned %v instead of expected pointer", idx, a) } if a := f(); a != nil { t.Errorf("Redundant removal %d returned %v instead of expected nil", idx, a) } expected = expected[1:] actual := walkAll(list) verifyOrder(t, expected, actual) </ins> } if list.Length() != 0 { t.Errorf("Expected length: %d, but got: %d)", 0, list.Length()) } <del> verifyOrder(t, arrival, dequeued) </del> } func TestFIFORemoveFromFIFOFunc(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned .

Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to .

hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .


Question: <del> 2.1 </del> <ins> 2.2 </ins>

/kind bug Hello, actually this is a bug about not about k8s. I use to observe request headers and found if the request have multiple same headers echoserver will give a wrong result: result: Expected result:
/sig testing
any chance you know some lua?
I never used lua before, but I'm willing to try to fix this issue.