Question: status.Ingress = []v1.LoadBalancerIngress{{IP: vip.Address}} <del> if lb.opts.FloatingNetworkId != "" { </del> <ins> if floatingPool != "" && !internalAnnotation { </ins> floatIPOpts := floatingips.CreateOpts{ <del> FloatingNetworkID: lb.opts.FloatingNetworkId, </del> <ins> FloatingNetworkID: floatingPool, </ins> PortID:            vip.PortID, } <ins> loadBalancerIP := apiService.Spec.LoadBalancerIP if loadBalancerIP != "" { floatIPOpts.FloatingIP = loadBalancerIP } </ins> floatIP, err := floatingips.Create(lb.network, floatIPOpts).Extract() if err != nil { return nil, fmt.Errorf("Error creating floatingip for openstack load balancer %s: %v", name, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: var r result client := newClient(true) for i := 0; i < 5; i++ { <del> r = doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 100*time.Millisecond) </del> <ins> r = doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 1*time.Second) </ins> if r.err != nil { break }

17:42 PDT 3/29 Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker. /sig testing
Looks like this is also being discussed and x-ref this test was merged 4 hours ago
/sig api-machinery
More x-ref: SIG api machinery is tracking it themselves in There is a PR to increase the timeout in
More x-ref: Same post in the .
Per this should be resolved and not be a blocker, testgrid is still red but I think the expectation is that it should clear up. Leaving this open for now but currently expecting this not to block release cut tomorrow.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "sync" "time" "github.com/prometheus/client_golang/prometheus" ) const ( kubeletSubsystem = "kubelet" ) var ( // HTTPRequests tracks the number of the http requests received since the server started. HTTPRequests = prometheus.NewCounterVec( prometheus.CounterOpts{ Subsystem: kubeletSubsystem, Name:      "http_requests_total", Help:      "Number of the http requests received since the server started", }, // server_type aims to differentiate the readonly server and the readwrite server. // long_running marks whether the request is long-running or not. // Currently, long-running requests include exec/attach/portforward/debug. []string{"method", "path", "host", "server_type", "long_running"}, ) // HTTPRequestsDuration tracks the duration in seconds to serve http requests. HTTPRequestsDuration = prometheus.NewHistogramVec( prometheus.HistogramOpts{ Subsystem: kubeletSubsystem, Name:      "http_requests_duration_seconds", Help:      "Duration in seconds to serve http requests", // Use DefBuckets for now, will customize the buckets if necessary. Buckets: prometheus.DefBuckets, }, []string{"method", "path", "host", "server_type", "long_running"}, ) // HTTPInflightRequests tracks the number of the inflight http requests. HTTPInflightRequests = prometheus.NewGaugeVec( prometheus.GaugeOpts{ Subsystem: kubeletSubsystem, Name:      "http_inflight_requests", Help:      "Number of the inflight http requests", }, []string{"method", "path", "host", "server_type", "long_running"}, ) ) var registerMetrics sync.Once // Register all metrics. func Register() { registerMetrics.Do(func() { prometheus.MustRegister(HTTPRequests) prometheus.MustRegister(HTTPRequestsDuration) prometheus.MustRegister(HTTPInflightRequests) }) } // SinceInSeconds gets the time since the specified start in seconds. func SinceInSeconds(start time.Time) float64 { return time.Since(start).Seconds() } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: # k8s echo "Download kubernetes release ..." <del> K8S_VERSION="v0.15.0" </del> <ins> K8S_VERSION="v0.18.0" </ins> if [ ! -f kubernetes.tar.gz ] ; then curl -L https://github.com/GoogleCloudPlatform/kubernetes/releases/download/${K8S_VERSION}/kubernetes.tar.gz -o kubernetes.tar.gz tar xzf kubernetes.tar.gz

This flag is deprecated and no longer has any effect.
I wonder which version does this include.Now ubuntu approach is using ubuntu 0.15.0.
This change was checked in this week, so it will be true for clusters at 0.18.0+ (once the next release is cut next week). So the next time the ubuntu documentation is updated, this change should be reflected.
Ohit means that minion node can be automatically discovered by the master node when then kubelet set ? No other configuration is needed?
Yes, the kubelet, when passed the location of the apiserver, will register itself and then post node update messages. The only part that may depend on your deployment environment is the set of addresses that the kubelet passes to the apiserver. They will be based on the name of the node, reverse resolving the name of the node, or checking for addresses from the cloud provider.
Got it.I will include this when I updates the doc and scripts. Besides, I have a little questions here. Eric told me the addon services do not have service-account now.Will 0.18.0+ have it?
Service accounts will be included as part of 0.18.0 as well.
got it
- makes sense


Question: // GetModules returns all installed kernel modules. func (handle *LinuxKernelHandler) GetModules() ([]string, error) { // Check whether IPVS required kernel modules are built-in <del> kernelVersionFile := "/proc/sys/kernel/osrelease" b, err := ioutil.ReadFile(kernelVersionFile) </del> <ins> kernelVersion, ipvsModules, err := utilipvs.GetKernelVersionAndIPVSMods(handle.executor) </ins> if err != nil { <del> glog.Errorf("Failed to read file %s with error %v", kernelVersionFile, err) </del> <ins> return nil, err </ins> } <del> kernelVersion := strings.TrimSpace(string(b)) </del> builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersion) <del> b, err = ioutil.ReadFile(builtinModsFilePath) </del> <ins> b, err := ioutil.ReadFile(builtinModsFilePath) </ins> if err != nil { glog.Warningf("Failed to read file %s with error %v. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", builtinModsFilePath, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: } // ReserveRegionAddress creates a region address <del> func (gce *GCECloud) ReserveRegionAddress(addr *compute.Address, region string) (*compute.Address, error) { </del> <ins> func (gce *GCECloud) ReserveRegionAddress(addr *compute.Address, region string) error { </ins> mc := newAddressMetricContext("reserve", region) op, err := gce.service.Addresses.Insert(gce.projectID, region, addr).Do() if err != nil { <del> return nil, mc.Observe(err) } if err := gce.waitForRegionOp(op, region, mc); err != nil { return nil, err </del> <ins> return mc.Observe(err) </ins> } <del> return gce.GetRegionAddress(addr.Name, region) </del> <ins> return gce.waitForRegionOp(op, region, mc) </ins> } // DeleteRegionAddress deletes a region address by name.

/kind bug What happened: Services of type LoadBalancer (GCE) with IP addresses promoted to static (by user or by orphan) cause the service controller to panic. The constant panicking will cause a reboot loop of the pod. How to reproduce it (as minimally and precisely as possible): a service on GCP it the GCP Console, promote the IP address of the LB to static and set the name to the loadBalancerName which is the same name as the forwarding rule. Environment: Kubernetes version: 1.7.0 AND 1.7.1 Cloud provider or hardware configuration: GCE Symptoms -controller-manager crash loops on the master. Mitigation** Option 1: Build your own version of branch for the master. Only the kube-controller-manager image needs to be updated. Option 2: For all affected services, set the field to the known external IP address. Steps: list of reserved IPs gcloud compute addresses list NAME REGION ADDRESS STATUS us-central1 35.35.35.35 INUSE list of all Services of type LoadBalancer kubectl get svc --all-namespaces | grep "LoadBalancer" default my-service LoadBalancer 10.0.99.201 35.35.35.35 80:/TCP,443:/TCP 58m each service of type LoadBalancer with an IP that's reserved (in the list above), continue to next step. the service's field equal to the known IP. Example_ apiVersion: v1 kind: Service metadata: creationTimestamp: 2017-07-13T00:09:31Z labels: app: echo name: my-service namespace: default resourceVersion: "" selfLink: /api/v1/namespaces/default/services/my-service uid: -675f-11e7-bee2- spec: clusterIP: 10.0.99.201 externalTrafficPolicy: Cluster loadBalancerIP: 35.35.35.35 <------- Field set to below. ports: name: port-1 nodePort: port: 80 protocol: TCP targetPort: 80 name: port-2 nodePort: port: 443 protocol: TCP targetPort: 443 selector: app: echo sessionAffinity: ClientIP type: LoadBalancer status: loadBalancer: ingress: ip: 35.35.35.35
/sig network /assign /area platform/gce
cc


Question: "custom_resource_definition.go", "discovery.go", "etcd_failure.go", <ins> "events.go", </ins> "framework.go", "garbage_collector.go", "generated_clientset.go",

[x] Original approval issue: [x] Two weeks soak end date: 6 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Events with a static label within a Namespace. that all 3 Events with the label in the Namespaces are created. the set of Namespaced Events with a label via DeleteCollection. that all Events with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen PR Merge and soaking
Reopened this issue. <detailsIn response to : /reopen PR Merge and soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func validateResourceClaimParametersRef(ref *resource.ResourceClaimParametersReference, fldPath *field.Path) field.ErrorList { var allErrs field.ErrorList <del> if ref != nil { if ref.Kind == "" { allErrs = append(allErrs, field.Required(fldPath.Child("kind"), "")) </del> <ins> if ref == nil { return allErrs } // group is required but the Core group is the empty value, so it can not be enforced. if ref.APIGroup != "" { for _, msg := range utilvalidation.IsDNS1123Subdomain(ref.APIGroup) { allErrs = append(allErrs, field.Invalid(fldPath.Child("apiGroup"), ref.APIGroup, msg)) } } if ref.Kind == "" { allErrs = append(allErrs, field.Required(fldPath.Child("kind"), "")) } else { for _, msg := range pathvalidation.IsValidPathSegmentName(ref.Kind) { allErrs = append(allErrs, field.Invalid(fldPath.Child("kind"), ref.Kind, msg)) </ins> } <del> if ref.Name == "" { allErrs = append(allErrs, field.Required(fldPath.Child("name"), "")) </del> <ins> } if ref.Name == "" { allErrs = append(allErrs, field.Required(fldPath.Child("name"), "")) } else { for _, msg := range pathvalidation.IsValidPathSegmentName(ref.Name) { allErrs = append(allErrs, field.Invalid(fldPath.Child("name"), ref.Name, msg)) </ins> } } return allErrs

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: // is supposed to have non-default value. // // TODO: Figure out which resource we should have non-default value. <ins> watchCacheSizes[Controllers] = maxInt(5*clusterSize, 100) </ins> watchCacheSizes[Endpoints] = maxInt(10*clusterSize, 1000) watchCacheSizes[Nodes] = maxInt(3*clusterSize, 1000) watchCacheSizes[Pods] = maxInt(10*clusterSize, 1000)

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
I think there are two different issues here - I thikn that I already debugged the second case. Basically, looking into logs, it seems that there are pretty big pauses between calls to apiserver from the test (my very strong hypothesis is that it is because of overloaded Jenkins machine where the test is running). These are the logs from the offending RC update: As a result, we will end up with "too old resource version" error coming from watch (since it was called ~5 seconds later then the get for it). However, this test is pretty big, so the solution for this problem is to actually make the size of the sliding window larger (we can afford it in large clusters). Will send a PR for it.
^^
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We should double check, but my hypothesis is that it may (similarly to our failures of kubemark-scale) be a consequence of overloaded Jenkins machine.
I checked the last failure and it's pretty obvious that the machine where the test is running is overloaded. We are calling this code: Get() and Update() are the method visible in apiserver logs. And here are the logs (corresponding get and put operations for one of RCs): As you can see, there are even ~20s breaks between those two consecutive calls. - FYI
FYI
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Hmm - we are now running tests on exclusive machines, and the symptoms are pretty much the same. The new hypothesis is that maybe client (from test) is being throttled?
Yeah - I confirmed running large kubemark on my own cluster, that the problem is actually throttling in the e2e test client. I will send out a PR that is fixing this.


Question: reason = fmt.Sprintf("ExitCode:%d", container.State.Terminated.ExitCode) } } else if container.Ready && container.State.Running != nil { <ins> hasRunning = true </ins> readyContainers++ } } <ins> // change pod status back to "Running" if there is at least one container still reporting as "Running" status if reason == "Completed" && hasRunning { reason = "Running" } </ins> } if pod.DeletionTimestamp != nil && pod.Status.Reason == node.NodeUnreachablePodReason {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When one pod is running and has two Containers. Container 1 exits with success and container 2 is running. gives correct info - the pod status is . However, , the pod status is . What you expected to happen: , the pod status is . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
i can not see your pod named nginx in your above code
, updated the typo.
/sig node /remove-sig api-machinery


Question: G []int             `json:"fg"` H []bool            `json:"fh"` I []float32         `json:"fi"` <ins> J []byte            `json:"fj"` </ins> } type G struct {

I have created the following secret in my cluster. When I add labels to my secret and try to apply it in the cluster I get the following error This also happens when I use or using the go client. Not sure why, but works. I expect that the secret can be updated with the new labels since the is not actually changed. Steps explained above. No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
/remove-sig cli /sig apps


Question: errs = append(errs, err) } } <ins> for _, iCtr := range pod.Spec.InitContainers { if err := safeToApplyPodPresetsOnContainer(&iCtr, podPresets); err != nil { errs = append(errs, err) } } </ins> return utilerrors.NewAggregate(errs) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /sig cluster-lifecyle What happened: I created a Pod Preset to inject an environment variable. The Pod container gets properly injected, but the init container does not. A small sample is here: What you expected to happen: The init container would get the pod preset injected How to reproduce it (as minimally and precisely as possible): Run kubectl create -f on the following gist: Reproduced here: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-09T07:26:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: minikube 0.23.0 OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Started with:
/sig cluster-lifecyle /sig service-catalog
My use case: I have an a git init container that checkout out configuration. This needs to run before the main container (hence why it is an init container). I'd like to be able to simply annotate the pod with a label and have various environment variables (GIT_REPO, etc) supplied to the init container
Update from Slack discussion: Supporting Init containers is in the Beta proposal for PodPreset. PodPreset resource is going to be moved from core to service-catalog. We have an implementation based on Initializers, but looks like Initializers are not going to Beta soon and we might have to reimplement them using Webhooks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
It'd be nice if this worked, I suspect it would help the folks over at Knative Build implement support for passing HTTP_PROXY env vars and CA cert secrets to build steps.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a customer interested in seeing this fixed - if we submit a PR, are there objections to merging it?
cc
This is also affecting me - I have an init container that loads data from AWS Secrets and stores them in a volume that is accessed by the main container. The init container needs credentials that are passed via a podpreset
any ETA when support for initcontainers will be provided by PodPreset ?
This was fixed in 1.14.


Question: } http := &http.Server{ <del> Addr:           rwLocation, </del> <ins> Addr:           wideOpenLocation, </ins> Handler:        apiserver.RecoverPanics(m.InsecureHandler), ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, MaxHeaderBytes: 1 << 20, } <del> glog.Infof("Serving insecurely on %s", rwLocation) </del> <ins> glog.Infof("Serving insecurely on %s", wideOpenLocation) </ins> glog.Fatal(http.ListenAndServe()) return nil }

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: It("should surface a failure condition on a common issue like exceeded quota", func() { rsConditionCheck(f) }) <ins> It("should adopt matching pods on creation", func() { testRSAdoptMatchingOrphans(f) }) It("should release no longer matching pods", func() { testRSReleaseControlledNotMatching(f) }) </ins> }) // A basic test to check the deployment of an image using a ReplicaSet. The

The merge queue is green with RBAC enabled, and the replicaset controller was missing permission to patch pods until fixed in Pod adoption and release depend on patching pods, which means that there are no merge-queue-blocking tests that test these features. Given that they are complex and rarely tested manually, we need tests that run as part of the merge queue to ensure they function correctly.
I'd like to work on this, where should the tests be ? I see that there are some patch-related tests for the replicaset controller in , and, according to http://submit-, the integration tests are run as part of the merge queue, is that correct? Thanks!
I think we'd want an e2e test that drives behavior externally on a normal cluster (by creating a replicaset, waiting to observe expected pods, modifies replicaset, waits to observe expected results, etc) . many of the integration tests tend to start up fake servers that don't share much in common with real deployments.
Ok thanks, on it.
cc:


Question: } return string(lastApplied), nil } <del> func isAnnotationsValid(annotations map[string]string) error { var totalSize int64 for k, v := range annotations { totalSize += (int64)(len(k)) + (int64)(len(v)) } if totalSize > (int64)(totalAnnotationSizeLimitB) { return fmt.Errorf("annotations size %d is larger than limit %d", totalSize, totalAnnotationSizeLimitB) } return nil } </del>

Expose totalAnnotationSizeLimitB in the validation package: This value is useful for validating annotations outside of the validation package, such as in SSA: Otherwise, size validation on annotations outside of the package would diverge and cause issues later. /kind cleanup /wg api-expression cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: **out = **in } <ins> if in.XValidations != nil { in, out := &in.XValidations, &out.XValidations *out = make([]ValidationRule, len(*in)) copy(*out, *in) } </ins> return out }

/kind failing-test /kind bug /sig api-machinery
/assign
hmm... latest run of the count10 job with this commit failed:
Interesting, that's a different TestRoundTrip? :eyes:
It seems to be getting lost here:
yeah... this is squirrely... that test is doing the following: internal types them to an openapi struct (which populates the extension pointing to internal structs) that to openapi json (which serializes Rule and Message in upper-case because there are no json tags on that struct) that to v1 JSONSchemaProps, which expects those fields to be and
I'll open another PR strengthening the test, and will tag in to figure out what the right thing to do here is this is the first structured extension we're adding that had subfields where capitalization mattered... I'm not 100% sure what we're doing here (serializing the internal types) is correct
PR in progress in
/triage accepted


Question: Generic Extensions <del> *ValueValidation </del> <ins> ValueValidation *ValueValidation </ins> } // +k8s:deepcopy-gen=true

What happened: API server panics handling requests for a CRD with OpenAPI validation with What you expected to happen: CRD is applied successfully. How to reproduce it (as minimally and precisely as possible): kind create cluster --image kindest/node:v1.16.1 then kubectl apply: Partial logs from API server: Anything else we need to know?: Environment: Kind Kubernetes version (use ):OS (e.g: ):
/sig api-machinery
/cc
where It seems that s.AnyOf was referencing some invalid address.
yes, which is very strange, given s was dereferenced earlier in the same method: is investigating
reproduced and has a fix
I had the same issue with prometheus (kube 1.16.1). Deploying is fine but when trying to delete, it crashes the api server. For those who can't upgrade to 1.16.2+ (fixed version) there is a workaround : using etcd to delete the resources that crash the api server : here I grep on coreos because I know it is prometheus that crashes the api server Remove every item in the list that causes the crash :
Note this is also resolved in 1.15.5+ and 1.16.2+, upgrading to 1.17.x is not required to resolve this


Question: featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/klog/v2" "k8s.io/klog/v2/ktesting" <ins> testingclock "k8s.io/utils/clock/testing" "k8s.io/utils/ptr" </ins> "k8s.io/kubernetes/pkg/features" schedulerapi "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/apis/config/testing/defaults"

/kind failing-test /kind flake First seen in an unrelated PR, but also occurs in periodic jobs. /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This feature is reverted once, and I fixed the flake, I'm cherry-pick that back again.


Question: case lowercaseCond == "create": return IsCreated, nil <del> case strings.HasPrefix(lowercaseCond, "condition="): conditionName := lowercaseCond[len("condition="):] </del> <ins> case strings.HasPrefix(condition, "condition="): conditionName := strings.TrimPrefix(condition, "condition=") </ins> conditionValue := "true" if equalsIndex := strings.Index(conditionName, "="); equalsIndex != -1 { conditionValue = conditionName[equalsIndex+1:]

In v1.31.0-rc.1 commands hang, see relevant thread for details. Second command should work, as it does with kubectl v1.30.3 Reproducer against a 1.30.3 server: No response <details</details<details</details<details</details<details</details<details</details<details</details
cc
looks like a regression introduced in /assign
thanks so much for the very detailed and timely report, great to catch this before the .0 release


Question: ENABLE_NODELOCAL_DNS=${KUBE_ENABLE_NODELOCAL_DNS:-false} DNS_SERVER_IP=${KUBE_DNS_SERVER_IP:-10.0.0.10} LOCAL_DNS_IP=${KUBE_LOCAL_DNS_IP:-169.254.20.10} <ins> DNS_MEMORY_LIMIT=${KUBE_DNS_MEMORY_LIMIT:-170Mi} </ins> DNS_DOMAIN=${KUBE_DNS_NAME:-"cluster.local"} KUBECTL=${KUBECTL:-"${KUBE_ROOT}/cluster/kubectl.sh"} WAIT_FOR_URL_API_SERVER=${WAIT_FOR_URL_API_SERVER:-60}

What happened: The following error occurs when creating kube-dns in local-up- What you expected to happen: hack/local-up- should be able to create kube-dns successfully. How to reproduce it (as minimally and precisely as possible): Run ./hack/local-up- Anything else we need to know?: The memory limit in kube- has been changed to "pillar['dnsmemorylimit']" by But local-up- does not transform it into a value. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing


Question: "${docker_extra_args[@]:+${docker_extra_args[@]}}"  "${METADATA_SERVER_ADD_HOST_ARGS[@]:+${METADATA_SERVER_ADD_HOST_ARGS[@]}}"  gcr.io/google_containers/kubekins-test:0.11  <del> bash -c "bash <(curl -fsS --retry 3 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/e2e-runner.sh')" </del> <ins> bash -c "bash <(curl -fsS --retry 3 --keepalive-time 2 'https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/e2e-runner.sh')" </ins>

http://pr- Seems like a flake? Could probably retry this, but not sure whether it's worth it.
we most definitely want to retry! Will you send a PR?
I just hit this in PR
We should close this, should be fixed by:
that didn't actually solve the problem. Per the docs, only works on "transient" errors, which the manpage defines as We might want to try passing , which some googling suggests might help. The default is 60s, which may be too long.
Please reactive this if you see it after June 15


Question: authInfoName, _ := config.getAuthInfoName() persister = PersisterForUser(config.configAccess, authInfoName) } <del> userAuthPartialConfig, err := getUserIdentificationPartialConfig(configAuthInfo, config.fallbackReader, persister) </del> <ins> userAuthPartialConfig, err := config.getUserIdentificationPartialConfig(configAuthInfo, config.fallbackReader, persister) </ins> if err != nil { return nil, err }

When I want to use basic-auth, kubectl will display the password I entered: I think it is more sensible to hide the password?
And it seems that kubectl request for username/password twice...
I'll take this. There is a library here: That appears to do the trick, but we need some refactoring so that it can read from an arbitrary reader to fit our usecase.
This was fixed by but I didn't close this issue. Closing it now.


Question: } }) <del> ginkgo.It("should apply changes to a resourcequota status", func() { </del> <ins> /* Release: v1.26 Testname: ResourceQuota, apply changes to a ResourceQuota status Description: Attempt to create a ResourceQuota for CPU and Memory quota limits. Creation MUST be successful. Updating the hard status values MUST succeed and the new values MUST be found. The reported hard status values MUST equal the spec hard values. Patching the spec hard values MUST succeed and the new values MUST be found. Patching the hard status values MUST succeed. The reported hard status values MUST equal the new spec hard values. Getting the /status MUST succeed and the reported hard status values MUST equal the spec hard values. Repatching the hard status values MUST succeed. The spec spec MUST NOT be changed when patching /status. */ framework.ConformanceIt("should apply changes to a resourcequota status", func() { </ins> ns := f.Namespace.Name rqClient := f.ClientSet.CoreV1().ResourceQuotas(ns) rqName := "e2e-rq-status-" + utilrand.String(5)

[X] Test approval issue: [X] Test PR: 19 Oct 2022 [x] Two weeks soak end date: 2 Nov 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining ResourceQuota status endpoints which are untested. patchCoreV1NamespacedResourceQuotaStatusreplaceCoreV1NamespacedResourceQuotaStatusreadCoreV1NamespacedResourceQuotaStatus Using a number of existing e2e test practices a new has been created for 3 ResourceQuota Status endpoints. The e2e logs for this test are listed below. This query shows which Resource Status endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return result } <ins> func isJobFinished(j *experimental.Job) bool { for _, c := range j.Status.Conditions { if c.Type == experimental.JobComplete && c.Status == api.ConditionTrue { return true } } return false } </ins> // byCreationTimestamp sorts a list by creation timestamp, using their names as a tie breaker. type byCreationTimestamp []experimental.Job

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: By("apply file doesn't have replicas") framework.RunKubectlOrDieInput(deployment2Yaml, "apply", "set-last-applied", "-f", "-", nsFlag) <del> By("check last-applied has been updated, annotations doesn't replicas") </del> <ins> By("check last-applied has been updated, annotations doesn't have replicas") </ins> output = framework.RunKubectlOrDieInput(deployment1Yaml, "apply", "view-last-applied", "-f", "-", nsFlag, "-o", "json") requiredString = ""replicas": 2" if strings.Contains(output, requiredString) { <del> framework.Failf("Missing %s in kubectl view-last-applied", requiredString) </del> <ins> framework.Failf("Presenting %s in kubectl view-last-applied", requiredString) </ins> } By("scale set replicas to 3")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: At e2e test for kubectl above, success scenario is key is not contained in result of . But test failure message is , in spite of key is contained. What you expected to happen: When test for absence of key failed, correct message is . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:none, hack/local-up- OS (e.g. from /etc/os-release):16.04.5 LTS (Xenial Xerus) Kernel (e.g. ):4.4.0-124-generic Install tools:hack/local-up- Others:
/sig cli


Question: return fmt.Errorf("failed to create log directory %q: %w", logPath, err) } // Copy logs (if any) to artifacts/hostname <del> if _, err := SSH(host, "ls", fmt.Sprintf("%s:%s/results/*.log", GetHostnameOrIP(host), testDir)); err == nil { </del> <ins> if _, err := SSH(host, "ls", fmt.Sprintf("%s/results/*.log", testDir)); err == nil { </ins> if _, err := runSSHCommand(host, "scp", "-r", fmt.Sprintf("%s:%s/results/*.log", GetHostnameOrIP(host), testDir), logPath); err != nil { return err }

As we are debugging nodee2e tests, we realize that we are missing kubelet logs and services. Kubelet logs allow us to see problems with kubelet which would be useful for all nodee2e jobs. Services are the other components that are started. I saw some warnings related to startup of apiserver so it would be nice to have these logs to troubleshoot.
/sig node
/assign
/triage accepted /priority important-longterm was this changed recently? I remember having seen not so long ago
It depends what tests you are looking at. The Kind e2e one do have but I didn't see any in the node_e2e ones. I'll do some research into that.
you are correct. Some of the jobs do have I am still trying to find out why a select few tests have a lot more logs than others. One thing I am noticing is that the jobs that are migrated to the community cluster seem to have much fewer logs. For example, is not yet migrated to the community cluster and it looks like we get a bunch of logs. Job Config: is migrated to community cluster and I don't see any extra logs here. Job Config:
/assign


Question: return RunKubectl(ns, "exec", fmt.Sprintf("--namespace=%v", ns), name, "--", "/bin/sh", "-x", "-c", cmd) } <ins> // RunHostCmdWithFullOutput runs the given cmd in the context of the given pod using `kubectl exec` // inside of a shell. It will also return the command's stderr. func RunHostCmdWithFullOutput(ns, name, cmd string) (string, string, error) { return RunKubectlWithFullOutput(ns, "exec", fmt.Sprintf("--namespace=%v", ns), name, "--", "/bin/sh", "-x", "-c", cmd) } </ins> // RunHostCmdOrDie calls RunHostCmd and dies on error. func RunHostCmdOrDie(ns, name, cmd string) string { stdout, err := RunHostCmd(ns, name, cmd)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ]

externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ]

externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


Question: } } switch { <ins> case waiting > 0: // One or more containers has not been started return api.PodPending, nil </ins> case running > 0 && unknown == 0: <ins> // All containers have been started, and at least // one container is running </ins> return api.PodRunning, nil case running == 0 && stopped > 0 && unknown == 0: <del> return api.PodFailed, nil case running == 0 && stopped == 0 && unknown > 0: return api.PodPending, nil </del> <ins> // All containers are terminated if pod.Spec.RestartPolicy.Always != nil { // All containers are in the process of restarting return api.PodRunning, nil } if stopped == succeeded { // RestartPolicy is not Always, and all // containers are terminated in success return api.PodSucceeded, nil } if pod.Spec.RestartPolicy.Never != nil { // RestartPolicy is Never, and all containers are // terminated with at least one in failure return api.PodFailed, nil } // RestartPolicy is OnFailure, and at least one in failure // and in the process of restarting return api.PodRunning, nil </ins> default: return api.PodPending, nil }

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: } return false } <ins> // normalizeURLString returns the normalized string, or an error if it can't be parsed into an URL object. // It takes an URL string as input. func normalizeURLString(s string) (string, error) { u, err := url.Parse(s) if err != nil { return "", err } if len(u.Path) > 0 { u.Path = strings.ReplaceAll(u.Path, "//", "/") } return u.String(), nil } </ins>

This is a proactive issue opened to explore the path forward with the module that we have been using as a dependency for in here This module is looking for a new maintainer. Discussions around the path forward, potentially eliminate the module from usage and do this operation in-house, as we use it in only one place for specific functionality. Ref Issue from repo: Slack Thread:
in kubeadm the usage of : Unicode normalization on the URL host according to "Unicode Normalization Form C". Unicode transformation to "fold" characters to canonical width on the host. the host from Unicode to ASCII (IDNA encoding, but 1 and 2 and needed due to a non-conformant x/net/idna at the time?) duplicate "//" from the URL path. the whole URL as per RFC 3986 (looks like PuerkitoBio/urlesc forks golang/net/url). IMO, we should: replace the usage of with this trimmed local version: then add a release note to the PR: we could import , and do the punycode conversion, but i do not understand why it should be kubeadm's responsibility to do that. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. i can see this being a breaking change to users having Unicode etcd endpoints in kubeadm, but i'm hoping the same users were already aware to not rely on k8s components to manage this for them.
/sig cluster-lifecycle /area kubeadm /remove-kind bug /kind cleanup
cool, that was helpful. I'll take up on this and create a PR.
/assign
From a Cluster API perspective: It's highly unlikely from the CAPI side that we'll specify IDNA in the kubeadm config. There is a chance on CAPV (vSphere) that a host picks up the DHCP option for the domain name suffix, and that it's using an IDNA. E.g., in my home environment, i have DHCP option 15 set to , and these end up in the etcd pods as parameters, e.g. Is IDNA encoding a consideration for input in etcd's command line, or are unicode args fine?

For the public clouds, it's even more unlikely we'll hit it as the choice of domain name suffix is highly constricted to meet node identity requirements in the CPI. My own take, is that I'm not wed to keeping this around at all, I haven't seen a use case for it.
cc and for any further vsphere comments.
have you seen DHCP with IDNA domains used for external etcd with kubeadm?
unicode args should be fine, also apparently Go's HTTP stack does IDNA encoding: i will update the proposed release note above. but IMO, if one would like a domain to be encoded when passed to all the tools in the k8s stack, maybe it should be done in advance. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. or leave it to the Go standard library to do the encoding. i saw comments that it might not be fully compliant yet, but i don't know how up to date these comments are.
/triage accepted i didn't see any PRs for this for 1.22. looks like we can update it in 1.23.
Apologies, I was a little occupied at work. But I have created the PR Please let me know if we can add this as part of 1.22.
no worries. we cannot add it in 1.22 because it's not a critical bug fix and we are in code freeze. i will comment on the PR.
looks like we should tell go-openapi/jsonreference to stop using the library too.
done. Created an issue in their repo as well and offered to create a PR if they are willing to move forward with in-sourcing the logic.
thanks!
just an FYI, the maintainers of the go-openapi have also removed the purell dependency.


Question: <del> 1.2 </del> <ins> 1.3 </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The image used in the Statefulset e2e tests: doesn't work because is missing in the image and the entrypoint is a bash script (). What you expected to happen: The image should be able to execute the entrypoint. How to reproduce it (as minimally and precisely as possible): Run the image with the same args as used in the manifests: Anything else we need to know?: Can fix the issue by adding to the install command: and building a new image: But I don't know if this is the right way since I don't fully understand the whole BASEIMAGE infrastructure and what's supposed to go where. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig testing


Question: <del> 1.2 </del> <ins> 1.3 </ins>

<!-- STOP -- PLEASE READ! GitHub is not the right place for support requests. If you're looking for help, check the or our . You can also post your question on the or the forum. If the matter is security related, please disclose it privately via --s390x support for nonewprivs image was in and the commit from was cherry picked in to fix agnhost broken on s390x. However, the post merge job failed for both( ) as image builder was broken. Now that image builder has been fixed in , these jobs need to be restarted to push s390x tags of agnhost and nonewprivs in staging bucket. cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
as you are on call this week, can you please run the CI job mentioned by hand?
, when I try to run the job with , it failed with . So I went over and looked at the history and found that this job was deleted by on 11/11. I don't think prow allows running a job that doesn't exist in its configmap, and searching the prow history with got: This is the list of jobs that have ever ran since prow keeps all history in GCS, which indicates that not all of the new jobs in has ran yet, can you take a look at and let us know which job is that you want?
thanks for looking. let's ask what jobs need to be triggered.
Hello, The post-kubernetes-push-e2e-test-images has been split into individual jobs for individual images and are now triggered by changes made in that image's folder. The jobs haven't been run yet since the split. You can trigger a job by sending a PR (e.g.: bumping the VERSION) and getting it merged.
/reopen Still failing according should fix it.
Reopened this issue. <detailsIn response to : /reopen Still failing according should fix it. Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close I am closing this issue as the agnhost job passed and the multiarch image is now available, it works fine for s390x as conformance job passed with the same
Closing this issue. <detailsIn response to : /close I am closing this issue as the agnhost job passed and the multiarch image is now available, it works fine for s390x as conformance job passed with the same Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // PortOpener is an interface around port opening/closing. // Abstracted out for testing. type PortOpener interface { <del> OpenLocalPort(lp *LocalPort) (Closeable, error) </del> <ins> OpenLocalPort(lp *LocalPort, isIPv6 bool) (Closeable, error) </ins> } // RevertPorts is closing ports in replacementPortsMap but not in originalPortsMap. In other words, it only

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: fs.IPVar(&s.BindAddress, "bind-address", s.BindAddress, ""+ "The IP address on which to listen for the --secure-port port. The "+ "associated interface(s) must be reachable by the rest of the cluster, and by CLI/web "+ <del> "clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).") </del> <ins> "clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.") </ins> desc := "The port on which to serve HTTPS with authentication and authorization." if s.Required {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver, controller-manager and scheduler are configured to use IPv4 addresses however, are serving on all addresses IPv4 and IPv6 What you expected to happen: To listen only on the IP family specified How to reproduce it (as minimally and precisely as possible): or checking in one of the gce presubmits jobs Anything else we need to know?: it seems that the apiserver, controller-manager and kube-scheduler listen on all IP families by default despite the IP family address configured Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /cc
binding 0.0.0.0 to all interfaces is the documented behavior of net.Listen:
The unspecified ip (0.0.0.0) is a special case that means "bind to all interfaces". This is currently working as intended. Is there a specific problem this causes?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If (0.0.0.0) means "bind to all interfaces on all IP families", then is a misconfiguration problem and not a bug . I was worried because this causes that some components use IPv6 for their communications on dual-stack hosts despite the cluster is IPv4 single family (and viceversa), as we can see in this snippet from one of the gce presubmits job and I don't know if this can impact on some services or from a security point of view
From the listen godoc: That means binding to or or will bind to all interfaces
I would expect anyone using network interfaces as security boundaries to be binding to a specific interface, not binding to all and hoping "all" is family-partitioned
Clarifying the handling of unspecified addresses like that in the flag/config help seems reasonable, but as far as I can tell, this is working as intended.

yeah, thanks for the clarification, I totally misinterpreted the net.Listen function
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
So, is there any way to listen on IPv4 addresses only (all of them)?
hmm, that will requires to expose this setting just curious, what is the use case for having IPv6 on the host and not in the apiserver, is not possible to disable ipv6 in the host?
I'm considering options to have a load balancer (HAProxy static pod in hostNetwork) for api-server on control plane nodes listening on the same port as api-server. Then I would point api-server VIP to via hosts file on the control plane nodes. This is a workaround for the absence of hairpin in AWS NLB (with instances by id) or Azure LB. I'm looking for a generic solution that will work for AWS/Azure/on-prem with nodes in the same/different subnets. If you have other suggestions, feel free to share, I would appreciate that. For now, my solution is to call , then patch the generated static manifest with set to the node's default ip, and then bootstrap with . Binding the load balancer to IPv6 and and api-server to IPv4, then using in kubeadm's config will be a dirty, but cleaner from the code perspective, approach. Btw, is there any reason behind having set to by default, but not the node's default ip, just like ?
I can't understand how do you solve the hairpin problem that way, is not the traffic been directed to the LB vip too? or do you mean that you don't use the AWS NLB and use the haproxy instead? what VIP do you use then? I don't remember the details, but the apiserver should be available to the pods too, that don't reach it through the node, they use the the kubernetes.default Service
External clients use AWS load balancer, while requests from control plane nodes go to haproxy on localhost thanks to trick, thus do not reach AWS LB.


Question: tags = ["automanaged"], deps = [ "//pkg/apis/rbac:go_default_library", <ins> "//pkg/features:go_default_library", </ins> "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apiserver/pkg/authentication/user:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/util/feature:go_default_library", </ins> ], )

/kind feature From: Would it make sense to add these rules to the bootstrappolicy? To me they seem generic enough, and they wouldn't bind to something. Consumers can do the binding to their subjects as they like. I'm asking because it's already duplicated twice for two early adopters: I don't think those ClusterRoles are specific to kubeadm and GKE at all; but the ClusterRoleBindings will surely be, so therefore we'd create the roles but not the bindings.
Wouldn't we expect and encourage people using the CSR capability for their nodes to use node identities that match the node authorizer, in which case we won't need the client or server renewal CSR roles?
The node authorizer doesn't grant these, since otherwise you could never prevent a compromised node from renewing its credentials
Realistically, with these roles you still wouldn't. People would grant them to the node group, not each individual node, but I guess that would be a tradeoff for the deployment mechanism.
You could individually revoke this role from the group and stop renewals without disrupting regular operation
I can definitely see myself revoking a binding from to multiple specific bindings in case of a compromise. Anyway, this issue is more "should we create these by default or not". I've already mentioned my opinion.
Creating the roles seems reasonable. I don't think I want bindings. I think the binding should be up the particular deployer.
Perfect, that's exactly my proposal. I'll go ahead and make this change...
/assign
are you working on this or should I send a PR?


Question: // FieldImmutableErrorMsg is a error message for field is immutable. const FieldImmutableErrorMsg string = `field is immutable` <del> const totalAnnotationSizeLimitB int = 256 * (1 << 10) // 256 kB </del> <ins> const TotalAnnotationSizeLimitB int = 256 * (1 << 10) // 256 kB </ins> // BannedOwners is a black list of object that are not allowed to be owners. var BannedOwners = map[schema.GroupVersionKind]struct{}{

Expose totalAnnotationSizeLimitB in the validation package: This value is useful for validating annotations outside of the validation package, such as in SSA: Otherwise, size validation on annotations outside of the package would diverge and cause issues later. /kind cleanup /wg api-expression cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: expectAssumedPod: podWithID("foo", testNode.Name), injectBindError:  errB, expectError:      errB, <del> expectErrorPod:   podWithID("foo", ""), </del> <ins> expectErrorPod:   podWithID("foo", testNode.Name), expectForgetPod:  podWithID("foo", testNode.Name), </ins> eventReason:      "FailedScheduling", }, { sendPod:     deletingPod("foo"),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: <ins> all: push TAG = 0.1 container: docker build -t gcr.io/google_containers/kubekins-test . docker tag gcr.io/google_containers/kubekins-test gcr.io/google_containers/kubekins-test:$(TAG) push: container gcloud docker push gcr.io/google_containers/kubekins-test # Push image tagged as latest to repository gcloud docker push gcr.io/google_containers/kubekins-test:$(TAG) # Push version tagged image to repository (since this image is already pushed it will simply create or update version tag) clean: </ins>

In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc


Question: CreateMirrorPod(pod *v1.Pod) error // DeleteMirrorPod deletes the mirror pod with the given full name from // the API server or returns an error. <del> DeleteMirrorPod(podFullName string) error </del> <ins> DeleteMirrorPod(podFullName string, uid *types.UID) (bool, error) </ins> } // basicMirrorClient is a functional MirrorClient.  Mirror pods are stored in

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod.

If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)?

I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!


Question: return fmt.Errorf("%s: Failed to list %v: %v", r.name, r.expectedType, err) } initTrace.Step("Objects listed") <del> r.metrics.listDuration.Observe(time.Since(start).Seconds()) </del> listMetaInterface, err := meta.ListAccessor(list) if err != nil { return fmt.Errorf("%s: Unable to understand list result %#v: %v", r.name, list, err)

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: the Pod is running. Pod SHOULD send a valid response when queried. release: v1.9 file: test/e2e/apps/replica_set.go <ins> - testname: ReplicaSet, status sub-resource codename: '[sig-apps] ReplicaSet should validate Replicaset Status endpoints [Conformance]' description: Create a ReplicaSet resource which MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. release: v1.22 file: test/e2e/apps/replica_set.go </ins> - testname: Replication Controller, adopt matching pods codename: '[sig-apps] ReplicationController should adopt matching pods on creation [Conformance]'

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return f.waitingPods.get(uid) } <ins> // RejectWaitingPod rejects a WaitingPod given its UID. func (f *framework) RejectWaitingPod(uid types.UID) { waitingPod := f.waitingPods.get(uid) if waitingPod != nil { waitingPod.Reject("removed") } } </ins> // HasFilterPlugins returns true if at least one filter plugin is defined. func (f *framework) HasFilterPlugins() bool { return len(f.filterPlugins) > 0

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: } if o.GracePeriod == 0 && !o.ForceDeletion { // To preserve backwards compatibility, but prevent accidental data loss, we convert --grace-period=0 <del> // into --grace-period=1 and wait until the object is successfully deleted. Users may provide --force // to bypass this wait. o.WaitForDeletion = true </del> <ins> // into --grace-period=1. Users may provide --force to bypass this conversion. </ins> o.GracePeriod = 1 } <del> if b, err := cmd.Flags().GetBool("wait"); err == nil { o.WaitForDeletion = b } </del> includeUninitialized := cmdutil.ShouldIncludeUninitialized(cmd, false) r := f.NewBuilder().

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc