Question: if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, "", nil } <del> return false, false, "", fmt.Errorf("error getting load balancer's forwarding rule: %v", err) </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, "", fmt.Errorf("error getting load balancer's forwarding rule: %v", err) </ins> } <del> if loadBalancerIP != fwd.IPAddress { </del> <ins> // If the user asks for a specific static ip through the Service spec, // check that we're actually using it. // TODO: we report loadbalancer IP through status, so we want to verify if // that matches the forwarding rule as well. if loadBalancerIP != "" && loadBalancerIP != fwd.IPAddress { glog.Infof("LoadBalancer ip for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.IPAddress, loadBalancerIP) </ins> return true, true, fwd.IPAddress, nil } portRange, err := loadBalancerPortRange(ports) if err != nil { <del> return false, false, "", err </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, "", err </ins> } if portRange != fwd.PortRange { <ins> glog.Infof("LoadBalancer port range for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.PortRange, portRange) </ins> return true, true, fwd.IPAddress, nil } // The service controller verified all the protocols match on the ports, just check the first one if string(ports[0].Protocol) != fwd.IPProtocol { <ins> glog.Infof("LoadBalancer protocol for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.IPProtocol, string(ports[0].Protocol)) </ins> return true, true, fwd.IPAddress, nil }

Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick


Question: op(queue, test.operands[i]) } <ins> queue.lock.Lock() </ins> for i := 0; i < len(test.expected); i++ { if pInfo, err := queue.activeQ.Pop(); err != nil { t.Errorf("Error while popping the head of the queue: %v", err)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: pkg/scheduler/internal/queue Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /assign
I just sent a pr for it according to the error message, but I'm not very sure if it will make things all right.
/assign


Question: return validateURL(issuerURL, disallowedIssuers, fldPath) } <del> func validateIssuerDiscoveryURL(issuerURL, issuerDiscoveryURL string, fldPath *field.Path) field.ErrorList { </del> <ins> func validateIssuerDiscoveryURL(issuerURL, issuerDiscoveryURL string, fldPath *field.Path, structuredAuthnFeatureEnabled bool) field.ErrorList { </ins> var allErrs field.ErrorList if len(issuerDiscoveryURL) == 0 { return nil } <ins> if !structuredAuthnFeatureEnabled { allErrs = append(allErrs, field.Invalid(fldPath, issuerDiscoveryURL, "discoveryURL is not supported when StructuredAuthenticationConfiguration feature gate is disabled")) } </ins> if len(issuerURL) > 0 && strings.TrimRight(issuerURL, "/") == strings.TrimRight(issuerDiscoveryURL, "/") { allErrs = append(allErrs, field.Invalid(fldPath, issuerDiscoveryURL, "discoveryURL must be different from URL")) }

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question: <ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Build the hyperkube base image. This image is used to build the hyperkube image. # # Usage: #   [ARCH=amd64] [REGISTRY="gcr.io/google-containers"] make (build|push) REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base TAG=0.1 ARCH?=amd64 CACHEBUST?=1 BASEIMAGE=gcr.io/google-containers/debian-base-$(ARCH):0.1 CNI_RELEASE=0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff TEMP_DIR:=$(shell mktemp -d) CNI_TARBALL=cni-$(ARCH)-$(CNI_RELEASE).tar.gz .PHONY: all build push clean all: push cni-tars/$(CNI_TARBALL): mkdir -p cni-tars/ cd cni-tars/ && curl -sSLO --retry 5 https://storage.googleapis.com/kubernetes-release/network-plugins/${CNI_TARBALL} clean: rm -rf cni-tars/ build: cni-tars/$(CNI_TARBALL) cp Dockerfile $(TEMP_DIR) cd $(TEMP_DIR) && sed -i "s|BASEIMAGE|$(BASEIMAGE)|g" Dockerfile ifeq ($(CACHEBUST),1) cd ${TEMP_DIR} && sed -i.back "s|CACHEBUST|$(shell uuidgen)|g" Dockerfile endif mkdir -p ${TEMP_DIR}/cni-bin tar -xz -C ${TEMP_DIR}/cni-bin -f "cni-tars/${CNI_TARBALL}" docker build --pull -t $(REGISTRY)/$(IMAGE)-$(ARCH):$(TAG) $(TEMP_DIR) rm -rf $(TEMP_DIR) push: build gcloud docker -- push $(REGISTRY)/$(IMAGE)-$(ARCH):$(TAG) </ins>

I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...


Question: }, "new serviceaccount with no secrets with resource conflict": { ClientObjects: []runtime.Object{updatedServiceAccount(emptySecretReferences()), createdTokenSecret()}, <ins> IsAsync:       true, MaxRetries:    1, </ins> AddedServiceAccount: serviceAccount(emptySecretReferences()), ExpectedActions: []core.Action{ core.NewGetAction(schema.GroupVersionResource{Version: "v1", Resource: "serviceaccounts"}, metav1.NamespaceDefault, "default"), <ins> core.NewGetAction(schema.GroupVersionResource{Version: "v1", Resource: "serviceaccounts"}, metav1.NamespaceDefault, "default"), </ins> }, }, "updated serviceaccount with no secrets": {

https://k8s- Failed: TestTokenCreation {serviceaccount} Previous issues for this test:
https://k8s- Failed: TestTokenCreation {serviceaccount}
fixed in (same fix as , missed that there was a second test involving resource conflicts)


Question: case *os.SyscallError: underlyingError = pe.Err } <del> return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE </del> <ins> return underlyingError == syscall.ENOTCONN || underlyingError == syscall.ESTALE || underlyingError == syscall.EIO </ins> } // GetSecretForPod locates secret by name in the pod's namespace and returns secret map

/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!


Question: []string{"operation"}, ) <ins> var vsphereVersion = metrics.NewDesc( "cloudprovider_vsphere_vcenter_versions", "Versions for connected vSphere vCenters", []string{"hostname", "version", "build"}, nil, metrics.ALPHA, "") </ins> // RegisterMetrics registers all the API and Operation metrics func RegisterMetrics() { legacyregistry.MustRegister(vsphereAPIMetric) legacyregistry.MustRegister(vsphereAPIErrorMetric) legacyregistry.MustRegister(vsphereOperationMetric) legacyregistry.MustRegister(vsphereOperationErrorMetric) <ins> legacyregistry.CustomMustRegister(vCenterMetric) } type vcenterMetric struct { metrics.BaseStableCollector mux          sync.Mutex vCenterInfos map[string]types.AboutInfo } func (collector *vcenterMetric) DescribeWithStability(ch chan<- *metrics.Desc) { ch <- vsphereVersion } func (collector *vcenterMetric) CollectWithStability(ch chan<- metrics.Metric) { collector.mux.Lock() defer collector.mux.Unlock() for vCenter, info := range collector.vCenterInfos { ch <- metrics.NewLazyMetricWithTimestamp(time.Now(), metrics.NewLazyConstMetric(vsphereVersion, metrics.GaugeValue, float64(1), vCenter, info.Version, info.Build)) } } func (collector *vcenterMetric) setAbout(server string, info types.AboutInfo) { collector.mux.Lock() defer collector.mux.Unlock() collector.vCenterInfos[server] = info } func setVCenterInfoMetric(connection *VSphereConnection) { vCenterMetric.setAbout(connection.Hostname, connection.Client.ServiceContent.About) </ins> } // RecordvSphereMetric records the vSphere API and Operation metrics

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: kube::release::package_kube_manifests_tarball & kube::util::wait-for-jobs || { kube::log::error "previous tarball phase failed"; return 1; } <del> kube::release::package_full_tarball & # _full depends on all the previous phases </del> <ins> kube::release::package_final_tarball & # _final depends on some of the previous phases </ins> kube::release::package_test_tarball & # _test doesn't depend on anything kube::util::wait-for-jobs || { kube::log::error "previous tarball phase failed"; return 1; } }

Either include non-arch specific stuff in every tar or break that out too. ref cc
interested in helping with this one?
Yes, I may help
OK, I did not actually understand what was in , apparently. Alternate proposal: [ ] update all references that need just server binaries to download the server tarball directly [ ] same for places that just need (related to ) [ ] remove all arch-specific binary artifacts from and replace with a script which downloads the client/server tarballs as necessary. The alternative option, including all of the non-arch-specific code and docs in an arch-specific starts to get very tricky, since we run into a large cross product of client x server platforms/archs.


Question: plugins       []*TestPlugin wantStatus    *framework.Status wantStatusMap framework.PluginToStatus <del> runAllFilters bool </del> }{ { name: "SuccessFilter",

Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers.

For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


Question: glog.Errorf("Unable to extract field %s: %s", fileInfo.FieldRef.FieldPath, err.Error()) errlist = append(errlist, err) } else { <del> fileProjection.Data = []byte(sortLines(values)) </del> <ins> fileProjection.Data = []byte(values) </ins> } } else if fileInfo.ResourceFieldRef != nil { containerName := fileInfo.ResourceFieldRef.ContainerName

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: } } <del> func (kl *Kubelet) generatePodStatus(pod *api.Pod, podStatus *kubecontainer.PodStatus) api.PodStatus { </del> <ins> func (kl *Kubelet) generateAPIPodStatus(pod *api.Pod, podStatus *kubecontainer.PodStatus) api.PodStatus { </ins> glog.V(3).Infof("Generating status for %q", format.Pod(pod)) // TODO: Consider include the container information. if kl.pastActiveDeadline(pod) {

We moved reason cache and status convert function from to by , , and . After this, we should also migrate related old unit test to kubelet level. For we should add test for: [x] Different image pull policy. [x] Pod creation failure. [x] Pod image failure. For we should add test for: [x] ContainerStatuses order. [x] Last termination state. [x] How reason cache will affect the PodPhase: [x] Pull image failure reason [x] Pod creation failure reason
Time to close this issue? :)
Not yet, The last 3 tests are in . :)


Question: } } <del> func AssertHistogramTotalCount(t *testing.T, name string, labelFilter map[string]string, wantCount int) { </del> <ins> func AssertHistogramTotalCount(t TB, name string, labelFilter map[string]string, wantCount int) { </ins> metrics, err := legacyregistry.DefaultGatherer.Gather() if err != nil { t.Fatalf("Failed to gather metrics: %s", err)

We have many "test only" packages through out the project, containing fixtures, helpers, utilities, etc... These test libraries are not intended for use in production binaries. They often lack rigorous review and sufficient testing for use in production binaries. We should make sure they don't accidentally get pulled in. Strawman proposal, test only packages must import "testing". Blank import is sufficient: Then write a presubmit that fails if we find symbol in the release binary. This approach is trivial and fairly effective although a more sophisticated approach could print a dep chain.
/assign
I hope I will be able to contribute once this gets marked as .
/triage accepted
I think that will be enough a new script or a check in the building scripts /cc since he has experience from the providerless building effort
I got what we want to achieve here but I'm little confused about: If I execute: after adding the import then only it is giving - what do we want to achieve by executing this command exactly? I got what it is doing. I've opened a thread in slack as well:
Hello! Is there anyone working on this issue at the moment? If not and the issue it's still relevant, I would like to work on it if possible. Thank you.
I opened a PR in the meantime, if it's not needed anymore, I can close it.


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "fmt" "math/rand" "testing" "time" "k8s.io/apimachinery/pkg/util/clock" compbasemetrics "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" "k8s.io/klog/v2" ) const ( samplesHistName       = "sawtestsamples" samplingPeriod        = time.Millisecond ddtRangeCentiPeriods  = 300 ddtOffsetCentiPeriods = 50 numIterations         = 100 ) /* TestSampler does a rough behavioral test of the sampling in a SampleAndWatermarkHistograms.  The test creates one and exercises it, checking that the count in the sampling histogram is correct at each step.  The sampling histogram is expected to get one observation at the end of each sampling period.  A fake clock is used, and the exercise consists of repeatedly changing that fake clock by an amount of time chosen uniformly at random from a range that goes from a little negative to somewhat more than two sampling periods.  The negative changes are included because small negative changes have been observed in real monotonic clock readings (see issue #96459) and we want to test that they are properly tolerated. The designed toleration is to pretend that the clock did not change, until it resumes net forward progress. */ func TestSampler(t *testing.T) { t0 := time.Now() clk := clock.NewFakePassiveClock(t0) buckets := []float64{0, 1} gen := NewSampleAndWaterMarkHistogramsGenerator(clk, samplingPeriod, &compbasemetrics.HistogramOpts{Name: samplesHistName, Buckets: buckets}, &compbasemetrics.HistogramOpts{Name: "marks", Buckets: buckets}, []string{}) saw := gen.Generate(0, 1, []string{}) regs := gen.metrics() for _, reg := range regs { legacyregistry.MustRegister(reg) } // `dt` is the admitted cumulative difference in fake time // since the start of the test.  "admitted" means this is // never allowed to decrease, which matches the designed // toleration for negative monotonic clock changes. var dt time.Duration // `t1` is the current fake time t1 := t0.Add(dt) klog.Infof("Expect about %v warnings about time going backwards; this is fake time deliberately misbehaving.", (numIterations*ddtOffsetCentiPeriods)/ddtRangeCentiPeriods) t.Logf("t0=%s", t0) for i := 0; i < numIterations; i++ { // `ddt` is the next step to take in fake time ddt := time.Duration(rand.Intn(ddtRangeCentiPeriods)-ddtOffsetCentiPeriods) * samplingPeriod / 100 t1 = t1.Add(ddt) diff := t1.Sub(t0) if diff > dt { dt = diff } clk.SetTime(t1) saw.Set(1) expectedCount := int64(dt / samplingPeriod) actualCount, err := getHistogramCount(regs, samplesHistName) if err != nil { t.Fatalf("For t0=%s, t1=%s, failed to getHistogramCount: %#+v", t0, t1, err) } t.Logf("For i=%d, ddt=%s, t1=%s, diff=%s, dt=%s, count=%d", i, ddt, t1, diff, dt, actualCount) if expectedCount != actualCount { t.Errorf("For i=%d, t0=%s, ddt=%s, t1=%s, expectedCount=%d, actualCount=%d", i, t0, ddt, t1, expectedCount, actualCount) } } } /* getHistogramCount returns the count of the named histogram */ func getHistogramCount(regs Registerables, metricName string) (int64, error) { considered := []string{} mfs, err := legacyregistry.DefaultGatherer.Gather() if err != nil { return 0, fmt.Errorf("failed to gather metrics: %s", err) } for _, mf := range mfs { thisName := mf.GetName() if thisName != metricName { considered = append(considered, thisName) continue } metric := mf.GetMetric()[0] hist := metric.GetHistogram() if hist == nil { return 0, fmt.Errorf("dto.Metric has nil Histogram") } if hist.SampleCount == nil { return 0, fmt.Errorf("dto.Histogram has nil SampleCount") } return int64(*hist.SampleCount), nil } return 0, fmt.Errorf("not found, considered=%#+v", considered) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Better testing of monotonic time in Why is this needed:
/sig api-machinery
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted </details
/triage accepted


Question: func TestExtractDeviceAndPrefix(t *testing.T) { devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00" <ins> mountPrefix := "/var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-default/" + devicePath </ins> lun := "-lun-0" <del> device, prefix, err := extractDeviceAndPrefix("/var/lib/kubelet/plugins/kubernetes.io/iscsi/" + devicePath + lun) if err != nil || device != (devicePath+lun) || prefix != devicePath { t.Errorf("extractDeviceAndPrefix: expected %s and %s, got %v %s and %s", devicePath+lun, devicePath, err, device, prefix) </del> <ins> device, prefix, err := extractDeviceAndPrefix(mountPrefix + lun) if err != nil || device != (devicePath+lun) || prefix != mountPrefix { t.Errorf("extractDeviceAndPrefix: expected %s and %s, got %v %s and %s", devicePath+lun, mountPrefix, err, device, prefix) } } func TestExtractIface(t *testing.T) { ifaceName := "default" devicePath := "127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0" iface, found := extractIface("/var/lib/kubelet/plugins/kubernetes.io/iscsi/iface-" + ifaceName + "/" + devicePath) if !found || iface != ifaceName { t.Errorf("extractIface: expected %s and %t, got %s and %t", ifaceName, true, iface, found) } iface, found = extractIface("/var/lib/kubelet/plugins/kubernetes.io/iscsi/" + devicePath) if found || iface != "" { t.Errorf("extractIface: expected %s and %t, got %s and %t", "", false, iface, found) </ins> } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): iscsi session logout Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Libvirt qemu VM, 4 CPU, 8Gig RAM OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): Linux localhost 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: What happened: If you have multiple pods on the same node using PVCs that are backed by iSCSI LUNs on the same external system over a single iSCSI session (same IQN but different LUN numbers), then deleting one of the pods causes the iSCSI session to terminate for all pods on that node that are using it. For example, here and are nginx pods each connected to a different PVC, which are backed by LUNs residing on the same NetApp storage system. These share a single iSCSI session, differing only by LUN number. Once pod2 is deleted, all the connected LUNs are no longer available.

[6:0:0:1] disk NETAPP LUN C-Mode 8300 /dev/sdc What you expected to happen: I expect that in-use iSCSI sessions are not terminated How to reproduce it (as minimally and precisely as possible): 2 PVs and PVCs that are backed by a single storage system that aggrates iSCSI connections into a single session. 2 pods that consume those PVCs one of the pods Anything else we need to know:
Could someone from please take a look? Thanks!
looking...
It looks like the problem is in line 86 of , in the function. The check evaluates whether a given mount path starts with the prefix (protocol + target portal + target IQN) for the LUN; this will always evaluate to false in most set-ups, since each path will look something like . Changing to appears to result in the correct behavior.
The problem is that the mount device shown in /proc/mounts are canonicalized, and thus when searching device prefix, iscsi volume failed to find a match. If we can use mount option, this the device will show up in /proc/mounts. But this is not a reliable fix since there are probable mpio paths too.
If I'm reading the code correctly, the function doesn't look at the actual device that's backing the mount point; it looks at the mount path (i.e., the second token in /proc/mounts, rather than the first). Thus, the change I proposed (HasPrefix -Contains) appears to work with some very basic testing. That said, I've not done anything with multipathing, so there may be something I'm missing; in addition, I suppose there's a chance that Contains may trigger false positives.
I see your point, yes that sounds right to me.
I think in this case, mpio device should work too (since the mount path constructed in the same way). I see you have yet another iscsi issue open, I am good with rescan. Can you create a pr to fix them?
I should be able create a PR for this issue; the LUN rescan PR will probably get handled by someone else on my team. Is the fix itself sufficient, or does it require tests?
thanks, the rescan fix doesn't have to unit test. The mount path fix needs also a unit test fix here

BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7; reproduced in Ubuntu 14.0.4.2. Kernel (e.g. ): Linux centos7 3.10.0-514.6.1.el7.x8664 SMP Wed Jan 18 13:06:36 UTC 2017 x8664 x8664 x8664 GNU/Linux; kernel 3.16.0-77-generic What happened: On a host where an iSCSI LUN from a given target is mounted in a pod, a user who creates a new LUN on the same target and attempts to create a pod that mounts the new LUN will fail to create the pod. Specifically, depending on the operating system, the pod will hang in the ContainerCreating stage with either a "Could not attach disk: Timeout after 10s" error (CentOS 7) or an error about establishing a session when one already exists (Ubuntu 14). What you expected to happen: The pod should successfully mount the LUN. How to reproduce it (as minimally and precisely as possible): Given a working iSCSI target and a single node Kubernetes cluster: a LUN on the iSCSI target a pod that uses the LUN from step 1 (either natively or through a PV/PVC) a new LUN on the iSCSI target (note that the target portal and IQN must be identical). a pod that uses the LUN from step 3. The pod created in step 4 will fail to start.

Anything else we need to know: As far as I can tell, this is due to a bug with the Kubernetes iSCSI implementation. The AttachDisk() function in first checks if the LUN exists in /dev/disk/by-path/; if it doesn't, it attempts to log in to the target and recheck for the LUN's existence (lines 123-138 in the 1.5 branch). In this case, though, the session already exists, as it was created with the pod in step 2. In Ubuntu, attempting to log in a second time triggers an error, which gets reported in the pod description. In CentOS 7, this gets treated as a NOP; however, the new LUN will never become visible, and thus, the pod will fail to start. This can be solved manually by logging into the host and running . It should be possible to fix it in Kubernetes by adding a check in AttachDisk() to see if a session already exists for the target portal and IQN and, if so, doing a rescan of the session.
Could someone from please help with triage? Thanks!


Question: } }, "spec": { <del> "replicas": 8, </del> <ins> "replicas": 2, </ins> "selector": { "name": "nettest" },

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.


Question: "reflect" "testing" <ins> "github.com/stretchr/testify/assert" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/api/testapi" <ins> "k8s.io/kubernetes/pkg/apis/core" </ins> api "k8s.io/kubernetes/pkg/apis/core" <ins> "k8s.io/kubernetes/pkg/apis/core/validation" </ins> "k8s.io/kubernetes/pkg/securitycontext" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a static pod for etcd as following with vsphere as cloud provider: But the hostname is using upper name, and when kubelet start the etcd static pod, it will report error as follows: I think that the vsphere should not return upper case hostname, but only lower case hostname. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area cloudprovider
Here is a quick
/sig node


Question: #   KUBE_USER #   KUBE_PASSWORD function get-password { <del> # templates to extract the username,password for the current-context user # Note: we save dot ('.') to $dot because the 'with' action overrides dot local username='{{$dot := .}}{{with $ctx := index $dot "current-context"}}{{$user := index $dot "contexts" $ctx "user"}}{{index $dot "users" $user "username"}}{{end}}' local password='{{$dot := .}}{{with $ctx := index $dot "current-context"}}{{$user := index $dot "contexts" $ctx "user"}}{{index $dot "users" $user "password"}}{{end}}' KUBE_USER=$("${KUBE_ROOT}/cluster/kubectl.sh" config view -o template --template="${username}") KUBE_PASSWORD=$("${KUBE_ROOT}/cluster/kubectl.sh" config view -o template --template="${password}") </del> <ins> get-kubeconfig-basicauth </ins> if [[ -z "${KUBE_USER}" || -z "${KUBE_PASSWORD}" ]]; then KUBE_USER=admin KUBE_PASSWORD=$(python -c 'import string,random; print "".join(random.SystemRandom().choice(string.ascii_letters + string.digits) for _ in range(16))')

Retrying after gave no joy.
Trying did not help.
will fix it. The problem is that your kubeconfig's current-context doesn't have a corresponding context entry. I'll see if I can make it gce's kube-up little more robust to invalid kubeconfig entries.
Thank you kindly for your help again. My brain is simply not large enough to remember all this.


Question: if proxier.healthzServer != nil { proxier.healthzServer.Updated() } <del> SyncProxyRulesLastTimestamp.SetToCurrentTime() </del> <ins> metrics.SyncProxyRulesLastTimestamp.SetToCurrentTime() </ins> // Update service healthchecks.  The endpoints list might include services that are // not "OnlyLocal", but the services list will not, and the serviceHealthServer

function on is not being used, and kproxy metrics are not showing on endpoint. See kube-proxy related metrics. Access kube-proxy on and search for kube-proxy specific metrics. No response <detailsKube-proxy build from master </details<details</details<details</details<details</details<details</details<details</details
/sig windows
/triage accepted /priority important-soon
/milestone v1.24


Question: Expect(err).NotTo(HaveOccurred()) } <ins> func setupLocalVolumeProvisionerMountPoint(config *localTestConfig, volumePath string) { By(fmt.Sprintf("Creating local directory at path %q", volumePath)) mkdirCmd := fmt.Sprintf("mkdir %v -m 777", volumePath) err := framework.IssueSSHCommand(mkdirCmd, framework.TestContext.Provider, config.node0) Expect(err).NotTo(HaveOccurred()) By(fmt.Sprintf("Mounting local directory at path %q", volumePath)) mntCmd := fmt.Sprintf("sudo mount --bind %v %v", volumePath, volumePath) err = framework.IssueSSHCommand(mntCmd, framework.TestContext.Provider, config.node0) Expect(err).NotTo(HaveOccurred()) } func cleanupLocalVolumeProvisionerMountPoint(config *localTestConfig, volumePath string) { By(fmt.Sprintf("Unmounting the test mount point from %q", volumePath)) umountCmd := fmt.Sprintf("sudo umount %v", volumePath) err := framework.IssueSSHCommand(umountCmd, framework.TestContext.Provider, config.node0) Expect(err).NotTo(HaveOccurred()) By("Removing the test mount point") removeCmd := fmt.Sprintf("rm -r %s", volumePath) err = framework.IssueSSHCommand(removeCmd, framework.TestContext.Provider, config.node0) } </ins> func createServiceAccount(config *localTestConfig) { serviceAccount := v1.ServiceAccount{ TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "ServiceAccount"},

The local PV provisioner is being updated to discover only real mount points (and not directories), that resolves some other related issues. The related e2e tests using the local PV provisioner need to be updated accordingly.
/assign
/sig storage
FYI:


Question: "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/validation" "k8s.io/apimachinery/pkg/util/validation/field" <ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" </ins> "k8s.io/apiserver/pkg/storage/value" kmstypes "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/kmsv2/v2alpha1" "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/metrics"

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: ContainersToStart: []int{1}, }, }, <ins> "Restart the container if the container is in created state": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.ContainerStatuses[1].State = kubecontainer.ContainerStateCreated }, actions: podActions{ SandboxID:         baseStatus.SandboxStatuses[0].Id, ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, ContainersToStart: []int{1}, }, }, </ins> } { pod, status := makeBasePodAndStatus() if test.mutatePodFn != nil {

It was consistently observed in a testing environment that a node-critical pod failed to start. event was present but was missing( is an init container). Also the following error was present but strangely only once: After some digging it seemed throwing an error matched the observations above. Eventually the init container and the pod to start The following tests in recreate the problem by and show how depending on the behavior is different. When the feature gate is enabled the last init container gets stuck in and never gets started. Therefore the pod get's stuck and never progresses The problem seems to stem from: There case get's skipped and the created container is expected to get started by the previous one by the following lines or But the provided tests show that this does not happen. Also if there is only one init container with the following logic should execute but won't because in this case this get's executed <details</details<details</details<details</details<details</details<details</details<details</details
/sig node can you help checking this issue report as according to it is a regression introduced with Thanks in advance!
I'll take a look. /assign
I guess this is a valid bug. /triage accepted /priority important-soon Can you provide a more specific scenario for this? Why do container runtime fail to start the container? (How can I reproduce it in e2e test?)
^ I guess this is the wrong assumption. If the container is in the state, the kubelet should resync it.
During node bootstrapping we had a bug where got restarted. So it is possible the following to happen: container gets created -restarts -container fails to start
We have a similar case. I have not yet looked at how containerd behaves at the moment, because it happens rarely and only at node startup. Pod stuck in the status. In the suppress status we have the following message: Alse we have the next events: Node characteristics: And it's like the error is reproduced on 1.29. The neighbouring cluster with k8s 1.28 is not affected by this problem
we have observed a similar behaviour but intermittently while upgrading our k8s cluster from k8s 1.29.0 to 1.29.6.

During upgrade process, when the node gets rebooted the "calico-node" pod of that node got stuck with below error forever until i restart the pod manually. Note: We are not observing this every time. Please let me know for any additional info required.
/cc


Question: return nil } <del> func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset utilipset.Interface, nodeIPs []net.IP, excludeCIDRs []string) *Proxier { </del> <ins> func NewFakeProxier(ipt utiliptables.Interface, ipvs utilipvs.Interface, ipset utilipset.Interface, nodeIPs []net.IP, excludeCIDRs []*net.IPNet) *Proxier { </ins> fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeCombinedOutputAction{ func() ([]byte, error) { return []byte("dummy device have been created"), nil },

What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping


Question: return r.store.Delete(ctx, name, options) } <ins> func shouldHaveOrphanFinalizer(options *metav1.DeleteOptions, haveOrphanFinalizer bool) bool { if options.OrphanDependents != nil { return *options.OrphanDependents } if options.PropagationPolicy != nil { return *options.PropagationPolicy == metav1.DeletePropagationOrphan } return haveOrphanFinalizer } func shouldHaveDeleteDependentsFinalizer(options *metav1.DeleteOptions, haveDeleteDependentsFinalizer bool) bool { if options.OrphanDependents != nil { return *options.OrphanDependents == false } if options.PropagationPolicy != nil { return *options.PropagationPolicy == metav1.DeletePropagationForeground } return haveDeleteDependentsFinalizer } </ins> func (e *REST) ConvertToTable(ctx context.Context, object runtime.Object, tableOptions runtime.Object) (*metav1beta1.Table, error) { return e.store.ConvertToTable(ctx, object, tableOptions) }

What happened: I created a namespace with finalizer, and then tried to delete it: The namespace can be deleted, and the finalizer is removed by garbage collection controller: What you expected to happen: Since kubectl species , it is not an orphaning operation, the finalizer should be removed in namespace store. How to reproduce it (as minimally and precisely as possible): Same as what I observed: a namespace. finalizer. . Anything else we need to know?: Environment: Kubernetes version (use ): HEAD
/sig api-machinery
cc
Discussion on slack:


Question: } // NodeAddresses is an implementation of Instances.NodeAddresses. <del> func (g *Cloud) NodeAddresses(_ context.Context, _ types.NodeName) ([]v1.NodeAddress, error) { internalIP, err := metadata.Get("instance/network-interfaces/0/ip") if err != nil { return nil, fmt.Errorf("couldn't get internal IP: %v", err) </del> <ins> func (g *Cloud) NodeAddresses(ctx context.Context, nodeName types.NodeName) ([]v1.NodeAddress, error) { timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour) defer cancel() instanceName := string(nodeName) if g.useMetadataServer { // Use metadata server if possible if g.isCurrentInstance(instanceName) { internalIP, err := metadata.Get("instance/network-interfaces/0/ip") if err != nil { return nil, fmt.Errorf("couldn't get internal IP: %v", err) } externalIP, err := metadata.Get("instance/network-interfaces/0/access-configs/0/external-ip") if err != nil { return nil, fmt.Errorf("couldn't get external IP: %v", err) } addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: internalIP}, {Type: v1.NodeExternalIP, Address: externalIP}, } if internalDNSFull, err := metadata.Get("instance/hostname"); err != nil { klog.Warningf("couldn't get full internal DNS name: %v", err) } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull}, v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull}, ) } return addresses, nil } </ins> } <del> externalIP, err := metadata.Get("instance/network-interfaces/0/access-configs/0/external-ip") </del> <ins> // Use GCE API instanceObj, err := g.getInstanceByName(instanceName) </ins> if err != nil { <del> return nil, fmt.Errorf("couldn't get external IP: %v", err) } addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: internalIP}, {Type: v1.NodeExternalIP, Address: externalIP}, </del> <ins> return nil, fmt.Errorf("couldn't get instance details: %v", err) </ins> } <del> if internalDNSFull, err := metadata.Get("instance/hostname"); err != nil { klog.Warningf("couldn't get full internal DNS name: %v", err) } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull}, v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull}, ) </del> <ins> instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(instanceObj.Name), instanceObj.Zone)) if err != nil { return []v1.NodeAddress{}, fmt.Errorf("error while querying for instance: %v", err) </ins> } <del> return addresses, nil </del> <ins> return nodeAddressesFromInstance(instance) </ins> } // NodeAddressesByProviderID will not be called from the node that is requesting this ID.

What would you like to be : Please see Implement Instances::NodeAddresses so it doesn't ignore the node input. Why is this needed: The current implementation assumes it will be only run on the Kubelet. That is no longer a valid assumption and in fact it the nearish future it will no longer be run on the Kubelet.
/sig cloud-provider /area provider/gcp
This issue is about the GCE implementation found at This problem cannot be fixed in the cloud node controller.
According to is indeed only used for the address of the calling instance. Maybe we should use instead? Note that implements
Or, we should definitely remove the if its TODO item is no longer true.
I believe the TODO is no longer true as there is also a call in


Question: // Adds an element to the start of the buffer (removing one from the end if necessary). func (self *TimedStore) Add(timestamp time.Time, item interface{}) { <del> // Remove any elements before the eviction time. evictTime := timestamp.Add(-self.age) index := sort.Search(len(self.buffer), func(index int) bool { return self.buffer[index].timestamp.After(evictTime) }) if index < len(self.buffer) { self.buffer = self.buffer[index:] } </del> // Remove any elements if over our max size. if self.maxItems >= 0 && (len(self.buffer)+1) > self.maxItems { startIndex := len(self.buffer) + 1 - self.maxItems self.buffer = self.buffer[startIndex:] } <ins> // Add the new element first and sort. We can then remove an expired element, if required. </ins> copied := item self.buffer = append(self.buffer, timedStoreData{ timestamp: timestamp, data:      copied, }) <ins> sort.Sort(self.buffer) // Remove any elements before eviction time. // TODO(rjnagal): This is assuming that the added entry has timestamp close to now. evictTime := timestamp.Add(-self.age) index := sort.Search(len(self.buffer), func(index int) bool { return self.buffer[index].timestamp.After(evictTime) }) if index < len(self.buffer) { self.buffer = self.buffer[index:] } </ins> } // Returns up to maxResult elements in the specified time period (inclusive).

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: if _, err := serviceaccount.ReadPrivateKey(f.Name()); err != nil { t.Fatalf("error reading private ECDSA key: %v", err) } <ins> if err := ioutil.WriteFile(f.Name(), []byte(ecdsaPrivateKeyWithParams), os.FileMode(0600)); err != nil { t.Fatalf("error writing private key to tmpfile: %v", err) } if _, err := serviceaccount.ReadPrivateKey(f.Name()); err != nil { t.Fatalf("error reading private ECDSA key with params: %v", err) } </ins> } func TestReadPublicKeys(t *testing.T) {

Kubernetes version Serverversion 1.5.2 (running Hyperkube v1.5.2coreos.1 ()) Environment: Bare-Metal Coreos 1235.6.0 (Kernel 4.7.3-coreos-r2) What happened: kube-controller-manager with ecdsa key fails to start F0127 09:59:04. 1 ] error running controllers: Error reading key for service account token controller: error reading private key file data does not contain a valid RSA or ECDSA private key The key is valid (and works with e.g. the api-server) Command used to create the key:
the issue is the block. removing that (invoke openssl with to omit it from the start) will allow the key to be consumed. will fix in 1.6 to ignore that block when reading the private key block


Question: "${SED}" -i '/CROSS_BUILD_/d' Dockerfile fi <del> docker buildx build --progress=plain --no-cache --pull --output=type="${output_type}" --platform "${os_name}/${arch}"  </del> <ins> # `--provenance=false --sbom=false` is set to avoid creating a manifest list: https://github.com/kubernetes/kubernetes/issues/123266 docker buildx build --progress=plain --no-cache --pull --output=type="${output_type}" --platform "${os_name}/${arch}" --provenance=false --sbom=false  </ins> --build-arg BASEIMAGE="${base_image}" --build-arg REGISTRY="${REGISTRY}" --build-arg OS_VERSION="${os_version}"  -t "${REGISTRY}/${image}:${TAG}-${suffix}" -f "${dockerfile_name}"  --label "image_version=${TAG}" --label "commit_id=${GIT_COMMIT_ID}"

agnhost was updated to v2.46 in: However, is failing: . exporting to image exporting layers exporting layers 2.9s done exporting manifest sha256:db8e9d1930d4799d474ea9228d4957d54053c107e10385d77021ef44f567fc69 done exporting config sha256:037153f61da4211d6ec5f00f694639ce15a0bbebf12fa5c86eea059fc3c0b07c done exporting attestation manifest sha256:744b616c276ff7f6518476765c2457e0b2664225ef798ce7e93874450807310b exporting attestation manifest sha256:744b616c276ff7f6518476765c2457e0b2664225ef798ce7e93874450807310b 0.0s done exporting manifest list sha256:29bf785c1a8338d5f7a8be8cae8a2435b94370eecbaa586a61e08454095940cb done pushing layers pushing layers 5.9s done pushing manifest for pushing manifest for 4.3s done DONE 13.2s /workspace/test/images /workspace/test/images/agnhost /workspace/test/images is a manifest list make: * [Makefile:43: all-build-and-push] Error 1 ERROR ERROR: build step 0 "" failed: step exited with non-zero status: 2
2024/02/13 15:34:04 Failed to run some build jobs: [error running [gcloud builds submit --verbosity info --config --substitutions PULLBASEREF=master,PULLBASESHA=,WHAT=agnhost,GITTAG=v20240213-v1.30.0-alpha.

1-173-gc3b2295c3a --project k8s-staging-e2e-test-images --gcs-log-dir gs://k8s-staging-e2e-test-images-gcb/logs --gcs-source-staging-dir gs://k8s-staging-e2e-test-images-gcb/source gs://k8s-staging-e2e-test-images-gcb/source/-7566-469c-a5a0-]: exit status 1] N/A 2024-02-14 No response No response_ /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR:


Question: serv.Flags |= utilipvs.FlagPersistent serv.Timeout = uint32(svcInfo.StickyMaxAgeSeconds()) } <ins> // Set the source hash flag needed for the distribution method "mh" if proxier.ipvsScheduler == "mh" { serv.Flags |= utilipvs.FlagSourceHash } </ins> // There is no need to bind Node IP to dummy interface, so set parameter `bindAddr` to `false`. if err := proxier.syncService(svcPortNameString, serv, false, bindedAddresses); err == nil { activeIPVSServices[serv.String()] = true

The current support of the load distribution method and in kube-proxy ipvs mode only performs hashing of source IP address to select endpoints. As a result, all connections from a single client (i.e. from a particular source IP) are always forwarded to only one endpoint matching with the hash value of this source IP. If the client opens a vast majority of connections to the k8s cluster, it will result in traffic imbalance as only one endpoint located on one node handles the connections while the remaining endpoints/nodes are idle. Here is an example of the problem when using method for kube-proxy Deploy a service with 4 endpoints located on 4 nodes.Open 200 connections from a client to node IP of any node (e.g. telnet 192.168.1.1 ). All connections receive replies from only one single endpoint.The distribution printout also shows that all connections are forwarded to the endpoint (e.g. 1.0.4.2:5001 in this example)Similar issue is seen with method Connections from a client should be evenly distributed to all endpoints like to avoid unbalanced load distribution between cluster nodes and endpoints. This can be easily done by enabling the source port hash flag that IPVS supports for the methods which performs hashing on both source IP and source port when selecting the target endpoint.
/sig network /area kube-proxy /area ipvs
This issue was raised based on the discussion of the issue
triage/accepted /assign
/triage/accepted
/triage accepted (damn it!)


Question: }, CapacityKb: int64(diskManager.volumeOptions.CapacityKB), } <ins> vdm := object.NewVirtualDiskManager(datastore.Client()) </ins> requestTime := time.Now() // Create virtual disk task, err := vdm.CreateVirtualDisk(ctx, diskManager.diskPath, datastore.Datacenter.Datacenter, vmDiskSpec)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In some cases, a create volume request can succeed in vSphere but time out in the vSphere storage plugin. This request will be retried, but fails with on subsequent attempts. As a result, the PersistentVolume object never gets created, and the pod requesting it never starts. Recovery requires manual steps. What you expected to happen: Since vSphere successfully created the volume, the storage plugin should reconcile and allow the PersistentVolume to get created. How to reproduce it (as minimally and precisely as possible): This is difficult to reproduce because it is caused by specific timing between vSphere and kubernetes. For testing, I a line of code to [][1] to inject failures after vSphere created the volume. [1]: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
Fixed by


Question: } EOF <del> # Configuration for k8s services that redirect logs to /var/log/<service>.log # files. Whenever logrotate is ran, this config will: </del> <ins> # Configure log rotation for all logs in /var/log, which is where k8s services # are configured to write their log files. Whenever logrotate is ran, this # config will: </ins> # * rotate the log file if its size is > 100Mb OR if one day has elapsed # * save rotated logs into a gzipped timestamped backup # * log file timestamp (controlled by 'dateformat') includes seconds too. This

GCI log rotation configuration missed "" file. We should just configure log rotation to rotate all "*.log" files in /var/log.
I cherrypicked your change last night. Please do what you need to do to confirm this is resolved.
Thank you, will do.
Verified. Closing issue.


Question: if err != nil { glog.Fatalf("Failed to make client: %v", err) } <ins> namespace := api.NamespaceSystem envNamespace := os.Getenv("NAMESPACE") if envNamespace != "" { if _, err := c.Namespaces().Get(envNamespace); err != nil { glog.Fatalf("%s namespace doesn't exist: %v", envNamespace, err) } namespace = envNamespace } </ins> var elasticsearch *api.Service // Look for endpoints associated with the Elasticsearch loggging service. // First wait for the service to become available. for t := time.Now(); time.Since(t) < 5*time.Minute; time.Sleep(10 * time.Second) { <del> elasticsearch, err = c.Services(api.NamespaceSystem).Get("elasticsearch-logging") </del> <ins> elasticsearch, err = c.Services(namespace).Get("elasticsearch-logging") </ins> if err == nil { break }

Now elasticsearchloggingdiscovery will only look for the endpoints in kube-system namespace due to hard-coded api.NamespaceSystem value. Maybe let user to decide in which namespace elasticsearchloggingdiscovery binary will look for endpoints ?


Question: } else { claim.Labels = set.Spec.Selector.MatchLabels } <del> claims[templates[i].Name] = claim </del> <ins> claims[templates[i].Name] = *claim </ins> } return claims }

The kube-controller-manage may crash when processing StatefulSet with spec.podManagementPolicy is Parallel. With logs: we expect the kube-controller-manage will work well. The following example can reproduce this problem: After digging into the source code, we found that when using the pod management policy, the statefulset controller will use to start parallel processing. But the which is called during parallel processing may directly update the claims' labels without a lock. This finally causes a concurrent map-accessing problem. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
Thanks for the report. Do we know when this issue was introduced? cc
Maybe it was imported by this pr
We can reproduce this issue in unit testing.
The reported versions is and it seems that PR got backported further /priority important-soon
/assign
The original report looks like could have introduced it, yes. I wonder if we intentionally modify template without creating a copy (and as it seems, assign the exact same labels every time we use this template for any pod): Regardless, that was probably safe before the pod creations were parallelized. I don't think the proposed unit test catches the bug (more on that below), but we can probably reproduce it by timing pod state transitions so that many pods are simultaneously pending and missing a PVC. I ran this test on (before the cherry-pick of parallel statefulset scaling), and it also failed. This is because the test itself introduces a concurrent map write. It probably doesn't replicate the issue, and I don't think it can be merged, as it'll fail even if the original PR is reverted. For reproduction we'd need to catch a concurrent write without adding one in test code, just as it happened based on the original report.
Turns out we never tested this path at all, since the test PVC didn't have any labels of its own.

After adding labels, existing tests are sufficient to detect race condition. PR showing that improved test detects the issue: PR with fix and improved test:
Sweet! thanks !


Question: result := c.client.clientBase.Put(). AbsPath(path). <del> Namespace(c.namespace). </del> <ins> NamespaceIfScoped(c.namespace, c.namespace != ""). </ins> Resource(gvr.Resource). Name(scale.Name). SubResource("scale").

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scale client () currently works fine with namespaced resources. However, it doesn't work for non-namespaced resources e.g. cluster-scoped CRDs. The current implementation should be extended in order to support non-namespaced resources. The current interface: /area custom-resources /sig autoscaling
I think scale subresource support for CRDs falls more into sig-apimachinery domain. After DirectXMan12 moved on from sig-autoscaling I don't think there is anyone in our sig-autoscaling with knowledge of scale subresource internals. /sig api-machinery
/assign
The bug report sounds legit. Does the generic scale client need to be changed (I think so), or does solve your issue?
No, the dashboard PR doesn't solve this. I discovered this issue while working on that PR so when gets merged, it will only work for namespaced CRDs. I think the generic scale client needs to be updated, maybe by adding another getter to the ScalesGetter interface: /kind bug
PR:


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/wait" <ins> applyadmissionregistrationv1 "k8s.io/client-go/applyconfigurations/admissionregistration/v1" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/openapi3" "k8s.io/kubernetes/test/e2e/framework"

<img width="1015" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
/priority critical-urgent Looking at the last green and first red shows the following: xref : /assign
hrmm, looks super solid in
yea something screwy going on with the kubeadm jobs :(
my main concern what that this was a test ..
kinder or kind jobs have been mostly solid across the board for a few years. if something is failing in kind / kinder then likely it would also fail in resource constraint scenarios. sorry but my 2c are that this is worth investigating by owning sigs
i don't know what else could be the cause here. kubeadm is crd agnostic.
completely agree this needs resolution, I'm just confused since is a really similar setup and looks completely solid The test waits until the status is set, so even a resource-constrained env should just wait a little longer ... the failures I see are bailing out of that loop after only one wait (105ms), so something is turning that field non-nil without populating the expected results I'm also pretty confused
the diff between the two jobs (kinder/kind) reveals that the kind job has much higher resource limits/requests (for CPU) full diff: <details</details
i'm not familiar with this area of the code. it does appear to be waiting for the TypeChecking struct to be populated, but not for the Warnings slice to be populated. logically (without trying to find the code): if TypeChecking is expected to be populated with Warnings during a Get call after the Create call, we are catching a flake bug. if the Warnings in the TypeChecking are populated async of the TypeChecking then we are catching a test flake and the test must be updated to wait for the Warnings too.
well, the code is here: it appears that the only way for this to happen is if the Check() call returns nil warnings under "some conditions". the problem is not reproducible in the unit test here if the interval is set to a lower value <= 100ms and tested with the tool. but that's with a fake client.

this extra err check seems redundant: it might be worth temporary adding a print of the crd and policy before this line: then allow this test to flake again and inspect the results.
/milestone v1.30
Thanks for raising the issue and thanks for the investigation! I don't know exactly why the solid test in kind would be flaky in kubeadm by a quick scan but agree it needs attention.(The tests look less flaky though..) /assign the type checking code owner. Would you mind adding a printout as suggested above and see if we could get more info? Thanks!
note, both kind and kinder (from the compared jobs) use kubeadm.
and i wonder if this is an informer cache issue.
Same guess as failed for missing event for service patch since 03/06(similar time). [Not sure; just guess]
shows that the test failed for a different reason. failure in ci-kubernetes-e2e-ec2-alpha-enabled-default: from 03/04 to 03/06.
/remove-kind failing-test /kind flake
the API field doc indicates a non-nil value means type checking is complete: It doesn't look like the type checker has any way to signal that it was lacking some information needed to complete type checking, and will record an empty set of warnings if it is: are there any scenarios where a laggy REST mapper or informer can make the controller think it completed type checking but be missing some info that makes it produce an empty list of warnings? I think our options are: the test to wait for non-empty warnings in the status (not just non-nil typeChecking) the controller to not write an empty typeChecking stanza if it fails to type check in some way option 2 seems better for observers, but I don't know how invasive that is
option 2 feels like we are creating a hole? (folks will see warnings sometimes and sometimes they don't?)
Option 2 might cause confusion especially when deleting existing warnings from typechecking field.

(We will remove the typechecking warning if observed the fix in schema)
I'm not suggesting deleting existing warnings, but it appears we are currently reporting an empty typeChecking stanza when we have not actually finished type checking... that doesn't match the API docs and is likely confusing to API clients (it's confusing our own e2e test we wrote based on the documented behavior)
Ignoring errors while finishing the type checking is intended behavior defined by the original KEP. Can we go back to change the tests to do a retry loop until the CRD appears in the RESTMapper? The root problem of laggy RESTMapper seems out of the scope of this test case.
if is fixing the test then it's masking a state not guaranteed by the field comment: i guess already explained that above.
The type checking is completed with this field set to non-nil. The problem is that the type checking finishes with error, and it cause some of the referred types to be ignored.
should this error be surfaced to the user and does that need an e2e test?
No. The KEP defines that errors will be ignored. However, there is a next step to make the type checker reacts on CRD changes, which handles the case when the CRD schema is late for the type checking.
i think users might be confused by this behavior (the same way people on this thread are) even if it's documented in a KEP, so a field comment update feels worth it, at minimum.
The current type checking is informational and only works on ValidatingAdmissionPolicy. If the CRD is created after type checking or CRD created but not showing in schema watcher yet, the error would be ignored. And it will not retry as long as ValidatingAdmissionPolicy remains unchanged. There is a PR for adding retry with schema resolver raised in which will solve the issue entirely. In current case, the guess would be schema watcher is slow so it didn't have crd info yet when performing typechecking. The temporary fix by in will resolve the test flaky by adding manually retry. The field comment of makes sense to me since it indicates the completion of the type checking but not handling the case where the CRD has not been created or created but not showing in schema watcher yet. FYI The says I agree it might not fit in conformance test. Not sure about the conformance test criteria.
cc
/triage accepted


Question: if len(elbPolicy.VIPs) == 0 || elbPolicy.VIPs[0] != vip { continue } <ins> } else if len(elbPolicy.VIPs) != 0 { continue </ins> } LogJson(plist, "Found existing Hns loadbalancer policy resource", 1) return &loadBalancerInfo{

What happened: I created a service where the was the same as the , and having done so, then the was not accessible externally on the Windows node where the pod was running (it was, however, accessible on the Linux master). When I changed either the or the so that they were different then the service was accessible on the Windows node. What you expected to happen: I expect services to work when the and the are the same value. How to reproduce it (as minimally and precisely as possible): Here's the file that I used: Then, in a web browser on my desktop machine, I visited the on both the Linux master's IP address, and then in a different tab, using the again, the IP address of the Windows worker node where that pod was scheduled. It only worked using the IP address of the Linux master. The tab using the Windows worker node's IP address did not load. Then, I changed either the or the so that they were different values, and when they were different, it worked using the of both the Linux master's IP address and the IP address of the Windows worker node where that pod was scheduled. Anything else we need to know?: On-premise install of k8s The Windows nodes all have the following feature gates enabled: WinOverlay WindowsGMSA Linux master: OS (e.g. ): Windows worker: If you need any more information then please let me know.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can you take a look?


Question: "fmt" "time" <ins> influxdb "github.com/influxdb/influxdb/client" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/kubernetes/pkg/api"

Jenkins project: kubernetes-e2e-gce-autoscaling failed last couple of runs: Initial Resources [Skipped][Autoscaling Suite] should set initial resources based on historical data Expected <int64: to be ~ <int:
couple failing instances:
Re-opening. This flaked again:
status?
The test is disabled and there is no plan to enable it within next few weeks, so closing the issue and I'll back to it some time in the future.


Question: "3": []algorithm.PredicateFailureReason{algorithmpredicates.ErrNodeUnderDiskPressure}, }, } <del> if strings.Contains(error.Error(), NoNodeAvailableMsg) { if strings.Contains(error.Error(), "NodeUnderDiskPressure (2)") && strings.Contains(error.Error(), "NodeUnderMemoryPressure (1)") { </del> <ins> if strings.Contains(err.Error(), NoNodeAvailableMsg) { if strings.Contains(err.Error(), "NodeUnderDiskPressure (2)") && strings.Contains(err.Error(), "NodeUnderMemoryPressure (1)") { </ins> return } } <del> t.Errorf("Error message doesn't have all the information content: [" + error.Error() + "]") </del> <ins> t.Errorf("Error message doesn't have all the information content: [" + err.Error() + "]") </ins> } // The point of this test is to show that you:

There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: utilerrors "k8s.io/kubernetes/pkg/util/errors" utilexec "k8s.io/kubernetes/pkg/util/exec" utiliptables "k8s.io/kubernetes/pkg/util/iptables" <ins> utilnet "k8s.io/kubernetes/pkg/util/net" </ins> utilsets "k8s.io/kubernetes/pkg/util/sets" utilsysctl "k8s.io/kubernetes/pkg/util/sysctl"

This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container.

The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again


Question: oldCfg, err := getCurrentKubeletConfig() framework.ExpectNoError(err) newCfg := oldCfg.DeepCopy() <ins> if newCfg.FeatureGates == nil { newCfg.FeatureGates = make(map[string]bool) } </ins> // Enable CPU Manager using feature gate. newCfg.FeatureGates[string(features.CPUManager)] = true

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Received panic when I make a e2e-node test: What you expected to happen: Do not panic. How to reproduce it (as minimally and precisely as possible): Execute test locally like: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing
/sig node


Question: API Server using the current Aggregator [Conformance]' description: Ensure that the sample-apiserver code from 1.17 and compiled against 1.17 will work on the current Aggregator/API-Server. <del> release: v1.17 </del> <ins> release: v1.17, v1.21 </ins> file: test/e2e/apimachinery/aggregator.go - testname: Custom Resource Definition Conversion Webhook, convert mixed version list codename: '[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin]

[x] test pr : kuberenetes/kubernetes# [x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [x] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. With this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Addressing the following endpoints in this document; Using the current e2e test as a template, create an APIService for Using the REST client, read the APIService status (readApiregistrationV1APIServiceStatus) and confirm the value of . Using the REST client, patch the APIService (patchApiregistrationV1APIService) before confirming that it has been updated. Using the REST client, list all APIServices in the cluster (listApiregistrationV1APIService) before confirming that can be found. Clean up APIService resources used by the test. Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. This query shows the endpoints hit within a short period of running the e2e test example endpoint | useragent ---------------------------------------|
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator2 (6 rows) Based on the initial APIsnoop query and the results listed above there are three new endpoints hit.

If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: } } <ins> type testNamespaceGetter struct { ns *corev1.Namespace } func (t *testNamespaceGetter) GetNamespace(ctx context.Context, name string) (*corev1.Namespace, error) { return t.ns, nil } </ins> type testPodLister struct { called bool pods   []*corev1.Pod

Tasks: [ ] status updates don't produce warnings [ ] spec updates don't block on enforce failures [ ] spec updates produce correct user-visible warnings
I want to do this and consult at slack.
/assign


Question: } func (jm *JobController) updateJobStatus(job *batch.Job) error { <del> _, err := jm.kubeClient.BatchV1().Jobs(job.Namespace).UpdateStatus(job) </del> <ins> jobClient := jm.kubeClient.BatchV1().Jobs(job.Namespace) var err error for i := 0; i <= statusUpdateRetries; i = i + 1 { var newJob *batch.Job newJob, err = jobClient.Get(job.Name, metav1.GetOptions{}) if err != nil { break } newJob.Status = job.Status if _, err = jobClient.UpdateStatus(newJob); err == nil { break } } </ins> return err }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: gvr:  myCRDV1Beta1, name: "sentinel-resource", patch: map[string]interface{}{ <del> // Just keep using different values sentinelName: fmt.Sprintf("invalid %v %v", uuidString, counter), </del> <ins> sentinelName: fmt.Sprintf("invalid-%d", counter), </ins> }}.Do(ctx) if err == nil {

integration TestRatchetingFunctionality/Enum TestRatchetingFunctionality/MinPropertiesMaxProperties TestRatchetingFunctionality/CELOptionalOldSelf TestRatchetingFunctionality/ TestRatchetingFunctionality/ since 3/8 No response /sig api-machinery
/assign /triage accepted