Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" . "github.com/onsi/ginkgo" ) var _ = Describe("Cadvisor", func() { var c *client.Client BeforeEach(func() { var err error c, err = loadClient() expectNoError(err) }) It("cadvisor should be healthy on every node.", func() { CheckCadvisorHealthOnAllNodes(c) }) }) func CheckCadvisorHealthOnAllNodes(c *client.Client) { By("getting list of nodes") nodeList, err := c.Nodes().List() expectNoError(err) for _, node := range nodeList.Items { // cadvisor is not accessible directly unless its port (4194 by default) is exposed. // Here, we access '/stats/' REST endpoint on the kubelet which polls cadvisor internally. statsResource := fmt.Sprintf("api/v1beta1/proxy/minions/%s/stats/", node.Name) By(fmt.Sprintf("Querying stats from node %s using url %s", node.Name, statsResource)) _, err = c.Get().AbsPath(statsResource).Timeout(1 * time.Second).Do().Raw() expectNoError(err) } } </ins>

We found yesterday that cAdvisor wasn't being started properly in 0.9.0 and 0.9.1 due to an issue with manifest file format changes (, fixed in ). If any of our e2e tests even checked that cAdvisor was running or exposing metrics on all nodes, it would have been caught before release. We should add at least some coverage for it to catch bugs like this.
If any of you are up for adding an e2e test for cadvisor, that'd be much appreciated!
I will work on adding one for cadvisor. , Alex Mohr wrote:
We already have a e2e test, which validate monitoring of Clusters using cAdvisor. We should figure out why the test passed without detecting such failure, and improve that test.
The existing test will succeed even if some of the nodes do not have cadvisor running. We can enhance the monitoring test to specifically check for cadvisor's health as well. , Dawn Chen wrote:
This test seems to be failing deterministically.
This was after fix to retry, but we're not exactly sure why.
Once I have emerged from the e2e test for cluster level logging with Elasticsearch quagmire (soon I hope -- perhaps by Wednesday) then I might also be willing to lend a hand with this since some of the experience might be relevant.
Actually latest error looks like So it's still not coming up after 5 minutes. The question is how long should it be taking to come up, given that the cadvisor containers get scheduled basically on node creation. The test finished in < 1s on my e2e cluster that has been up for awhile.
Or there's an actual bug here. can you look into it?
(I looked at the Jenkins log, and this test ran a good 10 minutes after startup, so it probably had 15 minutes to finish coming up. That seems wrong, given that cAdvisor is actually the first container we ever schedule, and it gets scheduled raw.)
Sorry, I hadn't seen -- not sure if that fixes this?
No, there are failures after as well. On Feb 20, 2015 5:39 PM, "David Oppenheimer" wrote:
I don't see any issue with the test itself. The test attempts to access a REST endpoint ('/stats') via the proxy on the api-server.

Is it possible that the api-server doesn't get to a stable state for a while?
This test executes after several others that hit the apiserver? On Feb 22, 2015 4:15 PM, "Vish Kannan" wrote:
Noob question: How can we tell if it runs this test after the other tests?
You can't tell deterministically (i.e. it's not hardcoded). I was merely commenting that in several runs where this fails, it has run after many other tests. We've been randomizing the spec order for a while. You can look at the Jenkins console logs to get the random seed for a given run: And the console logs have the order just by manually scanning the horizontal bars, but it's kind of pain. (That seed was off of go/k8s-test/job/kubernetes-e2e-gce/2638/consoleFull)
The reason for this failure is . I am closing this issue since the test itself is fine.


Question: BeforeEach(func() { c = f.ClientSet <ins> framework.Skipf("test is broken. #40249") </ins> nodelist := framework.GetReadySchedulableNodesOrDie(c) // Skip this test on small clusters.  No need to fail since it is not a use

The current way tags are implemented excludes scenarios where more than one tag is applicable. For instance [Flaky] supersedes [Disruptive], which leads to both disruptive and non-disruptive tests running together, causing flakes more in the non-disruptive. e.g. We are seeing a scenario where PersistentVolumes[Volume][Serial][Flaky] are run parallel to NodeOutOfDisk[Flaky][Disruptive]. NodeOutOfDisk causes the kubelet to evict pods, which leads to flakes in PersistentVolumes. Test grid @ https://k8s- What's the reasoning behind the current hierarchical tag scheme? Is there an inherent benefit over a multi-tag grouping system? For instance, one that could distinguish [Flaky][Disruptive] tests from [Flaky][Serial] ones? Or one that would run a [Flaky][Serial] test serially? (My understanding is that Flaky tests alway run in parallel.) Given that this is a stability issue for E2E, I'd like to ask that it be given a higher priority for a fix. cc
Ok. Agree it's high priority , for the fix keep this open, /'d maybe we need to put a pull request to the test infra repo? We need to find where the scripts that setup tests are, I don't know whether the Jenkins jobs that define what tests to skip/run are in kubernetes core anymore ...


Question: return true, nil } badPods = []api.Pod{} <ins> desiredPods = len(podList.Items) </ins> for _, pod := range podList.Items { if pod.Status.Phase != api.PodSucceeded { badPods = append(badPods, pod)

https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:


Question: controllerLogger = klog.LoggerWithName(controllerLogger, "ResourceSlice controller") controllerCtx = klog.NewContext(controllerCtx, controllerLogger) d.resourceSliceController = resourceslice.StartController(controllerCtx, d.kubeClient, d.driverName, owner, driverResources) <del> return </del> <ins> return nil </ins> } // Inform running controller about new information. d.resourceSliceController.Update(driverResources) <ins> return nil </ins> } // RegistrationStatus implements [DRAPlugin.RegistrationStatus].

When calling draplugin.Start without supplying kubeclient no error is reported, while the kubeclient is vital for further announcement of resourceSlice. An error needs to be returned when nodeName or kubeclient parameters are not set. Do not supply kubeclient parameter when creating draplugin, for isntance here: No response master <details</details<details</details<details</details<details</details<details</details
/sig node /triage accepted /project 95


Question: } if utilfeature.DefaultFeatureGate.Enabled(features.StatefulSetAutoDeletePVC) { <del> role.Rules = append(role.Rules, rbacv1helpers.NewRule("update").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie()) </del> <ins> role.Rules = append(role.Rules, rbacv1helpers.NewRule("update", "delete").Groups(legacyGroup).Resources("persistentvolumeclaims").RuleOrDie()) </ins> } return role

When OwnerReferencesPermissionEnforcement admission plugin is enabled, StatefulSetAutoDeletePVC feature cannot delete PVCs when scaling down a StatefulSet. StatefulSetAutoDeletePVC deletes PVCs when scaling down a StatefulSet Kubernetes with OwnerReferencesPermissionEnforcement admission & StatefulSetAutoDeletePVC feature gate enabled For example, using local-up- a StatefulSet with some replicas and: down the statefulset. PVCs are still present && complains: No response <detailsToday's master, v1.27.0-alpha.0.46+-dirty </details<details</details<detailsFedora </details<details</details<details</details<details</details
cc /sig storage /sig apps
/assign /triage accepted /priority important-soon


Question: func TestCalculateScaleDownLimitWithBehaviors(t *testing.T) { policy := autoscalingv2.MinChangePolicySelect <del> calculated := calculateScaleDownLimitWithBehaviors(5, []timestampedScaleEvent{}, &autoscalingv2.HPAScalingRules{ </del> <ins> calculated := calculateScaleDownLimitWithBehaviors(5, []timestampedScaleEvent{}, []timestampedScaleEvent{}, &autoscalingv2.HPAScalingRules{ </ins> StabilizationWindowSeconds: utilpointer.Int32Ptr(300), SelectPolicy:               &policy, Policies: []autoscalingv2.HPAScalingPolicy{

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // Indication the queue is closed. // Used to indicate a queue is closed so a control loop can exit when a queue is empty. // Currently, not used to gate any of CRED operations. <del> closed     bool closedLock sync.Mutex </del> <ins> closed bool </ins> // emitDeltaTypeReplaced is whether to emit the Replaced or Sync // DeltaType when Replace() is called (to preserve backwards compat).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: func (lr *LogReporter) SpecSuiteWillBegin(config config.GinkgoConfigType, summary *types.SuiteSummary) { b := &bytes.Buffer{} b.WriteString("******************************************************n") <del> glog.V(2).Infof(b.String()) </del> <ins> glog.V(0).Infof(b.String()) </ins> } func (lr *LogReporter) BeforeSuiteDidRun(setupSummary *types.SetupSummary) {}

Looks like it maybe the ssh connect is getting killed
The relevant log snippet:
And another one, not just coreos.


Question: // makeMounts determines the mount points for the given container. func makeMounts(pod *v1.Pod, podDir string, container *v1.Container, hostName, hostDomain, podIP string, podVolumes kubecontainer.VolumeMap) ([]kubecontainer.Mount, error) { <del> // Kubernetes only mounts on /etc/hosts if : // - container does not use hostNetwork and // - container is not an infrastructure(pause) container </del> <ins> // Kubernetes only mounts on /etc/hosts if: // - container is not an infrastructure (pause) container </ins> // - container is not already mounting on /etc/hosts <del> // When the pause container is being created, its IP is still unknown. Hence, PodIP will not have been set. // OS is not Windows mountEtcHostsFile := !pod.Spec.HostNetwork && len(podIP) > 0 && runtime.GOOS != "windows" </del> <ins> // - OS is not Windows // Kubernetes will not mount /etc/hosts if: // - when the Pod sandbox is being created, its IP is still unknown. Hence, PodIP will not have been set. mountEtcHostsFile := len(podIP) > 0 && runtime.GOOS != "windows" </ins> glog.V(3).Infof("container: %v/%v/%v podIP: %q creating hosts mount: %v", pod.Namespace, pod.Name, container.Name, podIP, mountEtcHostsFile) mounts := []kubecontainer.Mount{} for _, mount := range container.VolumeMounts { <ins> // do not mount /etc/hosts if container is already mounting on the path </ins> mountEtcHostsFile = mountEtcHostsFile && (mount.MountPath != etcHostsPath) vol, ok := podVolumes[mount.Name] if !ok || vol.Mounter == nil {

Currently, kubelet mounts hosts file for pods. For hostNetwork pods, the behavior is different and instead, kubelet relies on docker (or whatever container runtime) to mount the host's hosts file. We should have kubelet handle hosts file for hostNetwork pods. This would have several benefits: control over hosts file in case the container runtime does something consistent behavior between hostNetwork and non-hostNetwork pods consistent behavior between different container runtimes -managed hosts file enables features like to be used for hostNetwork pods Implementation would require: kubelet's or its caller output different content based on hostNetwork. i. If it's not hostNetwork, use existing behavior. ii. If it's using hostNetwork, read the content from the host's hosts file. hostNetwork constraint in deciding whether to mount the file Open Questions: do we treat changes to the host's hosts file? should the file's content be static between container restarts of the same Pod? or should every container restart pick up the latest content of the hosts file? the former seems better for consistency, but letter might be more in line with other features that use host resources like host volumes. condition to manage hosts file also excludes windows even though windows also the use of hosts file and the same format. seems like this should be supported also. windows-support ships, should it have validation against windows platform? the tricky bit is the client, API, and minions' kubelet may be running on different platforms, and currently, there's no precedence to platform validation. /kind feature /sig node


Question: if [[ "${NETWORK_PROVIDER:-}" == "kubenet" ]] ||  [[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then #TODO(andyzheng0831): We should make the cni version number as a k8s env variable. <del> local -r cni_tar="cni-8a936732094c0941e1543ef5d292a1f4fffa1ac5.tar.gz" </del> <ins> local -r cni_tar="cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz" </ins> download-or-bust "" "https://storage.googleapis.com/kubernetes-release/network-plugins/${cni_tar}" local -r cni_dir="${KUBE_HOME}/cni" mkdir -p "${cni_dir}"

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: } for _, image := range containerImages { <del> names := append(image.RepoDigests, image.RepoTags...) </del> <ins> // make a copy to avoid modifying slice members of the image items in the list names := append([]string{}, image.RepoDigests...) names = append(names, image.RepoTags...) </ins> // Report up to MaxNamesPerImageInNodeStatus names per image. if len(names) > MaxNamesPerImageInNodeStatus { names = names[0:MaxNamesPerImageInNodeStatus]

See Today kubelet image manager returns the internal image list cache to caller directly. However, in we sort the image list, which introduces a data race. We should let image manager return a copy of image list instead of returning the list directly. /cc
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code Freeze for 1.10 begins in 2 hours. If this issue represents a bug or change for 1.10, then it needs SIG approval in the form of a status/approved-for-milestone label. It also needs a priority/ Thanks!


Question: glog.V(4).Infof("Pods to delete for daemon set %s: %+v", ds.Name, podsToDelete) for i := range podsToDelete { <del> if err := dsc.podControl.DeletePod(ds.Namespace, podsToDelete[i]); err != nil { </del> <ins> if err := dsc.podControl.DeletePod(ds.Namespace, podsToDelete[i], ds); err != nil { </ins> glog.V(2).Infof("Failed deletion, decrementing expectations for set %q/%q", ds.Namespace, ds.Name) dsc.expectations.DeletionObserved(dsKey) util.HandleError(err)

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc


Question: E2E_STORAGE_TEST_ENVIRONMENT=${KUBE_E2E_STORAGE_TEST_ENVIRONMENT:-false} # Evict pods whenever compute resource availability on the nodes gets below a threshold. <del> EVICTION_HARD="${EVICTION_HARD:-memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%}" </del> <ins> EVICTION_HARD="${EVICTION_HARD:-memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%}" </ins> # Optional: custom scheduling algorithm SCHEDULING_ALGORITHM_PROVIDER="${SCHEDULING_ALGORITHM_PROVIDER:-}"

Clusters on GCE set eviction thresholds for disk and inodes. Given that reclaiming disk and inodes is slow, we should set minimum reclaim thresholds too. Essentially, set . I assume percentages are supported for min reclaims too. Do we have system tests for min reclaim? cc
I just took a glance at the parser for eviction minimum reclaim, it doesn't look like it tries to parse percentages. can you confirm?
did percentage support, and we did not extend it to min reclaim. It's a good idea to do though. On Tuesday, September 13, 2016, Michael Taufen wrote:
Hmm.. In that case for v1.4, can we set the min reclaim to ? What values are you planning on using in Open Shift ?
This should fix this issue to avoid node condition being flip-flop, but I think it shouldn't block 1.4 release.
More than flip-flop, it will help the node recover from disk pressure condition sooner. , Dawn Chen wrote:
/cc
Spawned to add percentage support for min reclaim. I will follow up with our proposed default for nodefs later
This needs to be triaged as a release-blocker or not for 1.5
can you check if we have set default values for disk and inode based evictions in kubelet? If not, we should add them.
I believe our current default value for the --eviction-hard flag is "memory.available<100Mi", found in Our tests currently run using: "memory.available<250Mi,nodefs.available<10%,nodefs.inodesFree<5%", found in What should I change the default to? This issue is about minimum reclaim, should I look into changing those flags as well?
It doesn't appear that we have defaults for minimum reclaim, should I add minimum reclaim for memory in addition to disk and inodes?
I think GCE defaults do not apply globally, we should change what GCE does, and handle setting global defaults in 1.6.
the global defaults are expected to be based on our OSS tests. Our original plan was to roll out disk eviction by default in v1.5. This is not specific to GCE.
Can you make a call on if this is a blocker or non-blocker for 1.5 by EOD Friday please


Question: if fsVolume { // Filesystem volume case // Mount/remount a volume when a volume is attached <del> generatedOperations, err = oe.operationGenerator.GenerateMountVolumeFunc( </del> <ins> generatedOperations = oe.operationGenerator.GenerateMountVolumeFunc( </ins> waitForAttachTimeout, volumeToMount, actualStateOfWorld, isRemount) } else {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.


Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "flag" "fmt" "io/ioutil" "os" "syscall" ) var ( fsTypePath           = "" fileModePath         = "" readFileContentPath  = "" readWriteNewFilePath = "" ) func init() { flag.StringVar(&fsTypePath, "fs_type", "", "Path to print the fs type for") flag.StringVar(&fileModePath, "file_mode", "", "Path to print the filemode of") flag.StringVar(&readFileContentPath, "file_content", "", "Path to read the file content from") flag.StringVar(&readWriteNewFilePath, "rw_new_file", "", "Path to write to and read from") } // This program performs some tests on the filesystem as dictated by the // flags passed by the user. func main() { flag.Parse() var ( err  error errs = []error{} ) err = fsType(fsTypePath) if err != nil { errs = append(errs, err) } err = readWriteNewFile(readWriteNewFilePath) if err != nil { errs = append(errs, err) } err = fileMode(fileModePath) if err != nil { errs = append(errs, err) } err = readFileContent(readFileContentPath) if err != nil { errs = append(errs, err) } if len(errs) != 0 { os.Exit(1) } os.Exit(0) } // Defined by Linux (sys/statfs.h) - the type number for tmpfs mounts. const linuxTmpfsMagic = 0x01021994 func fsType(path string) error { if path == "" { return nil } buf := syscall.Statfs_t{} if err := syscall.Statfs(path, &buf); err != nil { fmt.Printf("error from statfs(%q): %v", path, err) return err } if buf.Type == linuxTmpfsMagic { fmt.Printf("mount type of %q: tmpfsn", path) } else { fmt.Printf("mount type of %q: %vn", path, buf.Type) } return nil } func fileMode(path string) error { if path == "" { return nil } fileinfo, err := os.Lstat(path) if err != nil { fmt.Printf("error from Lstat(%q): %vn", path, err) return err } fmt.Printf("mode of file %q: %vn", path, fileinfo.Mode()) return nil } func readFileContent(path string) error { if path == "" { return nil } contentBytes, err := ioutil.ReadFile(path) if err != nil { fmt.Printf("error reading file content for %q: %vn", path, err) return err } fmt.Printf("content of file %q: %vn", path, string(contentBytes)) return nil } func readWriteNewFile(path string) error { if path == "" { return nil } content := "mount-tester new filen" err := ioutil.WriteFile(path, []byte(content), 0644) if err != nil { fmt.Printf("error writing new file %q: %vn", path, err) return err } return readFileContent(path) } </ins>

I'd like to commit without e2e support so others can unblock. We need to come back to get e2e on this. I could use help, and it seems like an easy test...
Sorry for my incorrect PR. Now I have amend it.


Question: Name:    "pv-recycler", Image:   "registry.k8s.io/debian-base:v2.0.0", Command: []string{"/bin/sh"}, <del> Args:    []string{"-c", "test -e /scrub && rm -rf /scrub/..?* /scrub/.[!.]* /scrub/*  && test -z "$(ls -A /scrub)" || exit 1"}, </del> <ins> Args:    []string{"-c", "test -e /scrub && find /scrub -mindepth 1 -delete && test -z "$(ls -A /scrub)" || exit 1"}, </ins> VolumeMounts: []v1.VolumeMount{ { Name:      "vol",

PV recycler pod fails to remove files if there are files in the directory. Error in the pod logs: Apparently one hits ARG_MAX due to too long command line. PV is recycled successfully Claim and NFS pv, create files. In my testing I have provided my template and replaced the with: and recycle was successful. Maybe I am missing something but the was able to clean up everything (files, folders) and we also need to exclude the folder name. already has the command, no changes to image needed. <details</details<detailsBare metal </details<details</details<details</details<details</details<detailsNFS Persistent Volume </details
/sig storage
I can prep a PR if we are happy with using command.
/triage accepted
/help wanted
Feel free to submit a PR.
updated the command in issue description as per


Question: var CmdFakeGitServer = &cobra.Command{ Use:   "fake-gitserver", Short: "Fakes a git server", <del> Long: `When doing "git clone localhost:8000", you will clone an empty git repo named "8000" on local. You can also use "git clone localhost:8000 my-repo-name" to rename that repo.`, </del> <ins> Long: `When doing "git clone http://localhost:8000", you will clone an empty git repo named "localhost" on local. You can also use "git clone http://localhost:8000 my-repo-name" to rename that repo.`, </ins> Args: cobra.MaximumNArgs(0), Run:  main, }

What happened: Follow the "README", I hit below issue, the command there need a prefix "http" or "https". What you expected to happen: clone a fake repo successfully. How to reproduce it (as minimally and precisely as possible): follow the "README" here. create a pod create a service git clone localhost:8000 Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all
/assign
/sig testing
/kind documentation


Question: RouteTableName:               "rt", PrimaryAvailabilitySetName:   "as", MaximumLoadBalancerRuleCount: 250, <ins> VMType: vmTypeStandard, </ins> }, } az.DisksClient = newFakeDisksClient()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When using IMDS, kubelet reports even when vmtype is not set: What you expected to happen: azure cloud provider should assume stardard vmtype if it is not set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
Added to v1.10 milestone since it is a critical bug fix. cc


Question: ChainPrerouting  Chain = "PREROUTING" ChainOutput      Chain = "OUTPUT" ChainInput       Chain = "INPUT" <ins> ChainBrouting    Chain = "BROUTING" </ins> ) type operation string

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!


Question: pathOptions.EnvVar = "" buf := bytes.NewBuffer([]byte{}) <del> cmd := NewCmdConfigDeleteContext(buf, pathOptions) </del> <ins> errBuf := bytes.NewBuffer([]byte{}) cmd := NewCmdConfigDeleteContext(buf, errBuf, pathOptions) </ins> cmd.SetArgs([]string{test.contextToDelete}) if err := cmd.Execute(); err != nil { t.Fatalf("unexpected error executing command: %v", err)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, this is a bug report. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: MacBook Pro Mid-2015 OS (e.g. from /etc/os-release): macOS Kernel (e.g. ): Install tools: Others: N/A What happened: When deleting the context that was my current context, I entered a state where I am seemingly unable to set my context. The shows none as the current: Also, when trying to take any actions that require a context the following error is presented: When viewing the full config () I can see that my selected context is still the one I deleted. What you expected to happen: I believe there are two expectations I had: should allow me to delete my current context, but not enter an inconsistent state in the process -or- should either provide a warning or disallow removing your current context How to reproduce it (as minimally and precisely as possible): I believe that you just need a configuration with multiple contexts. I had three before deleting the context that has me unable to set a new context.

list all contexts to find your current context () delete your current context list your current contexts again, and you'll see no contexts are created try to set your context with one of the ones that exist () listing your contexts again will show none being set Anything else we need to know: 3 contexts were configured before starting to delete one 2 of them were GKE clusters, automatically set up by the binary 1 is a context set up by the context I deleted was one of the contexts The issue was resolved by manually setting my context by editing .
I think this is the confusion of doesn't switch contexts, it lets you set fields on a context. is what switches contexts.
It's technically valid to have no current context, but I agree a warning message like "warning: this removed your active context, use to select a different one" would go a long way.
SGTM.


Question: fakeDocker.Unlock() } <del> func TestGarbageCollectImages(t *testing.T) { kubelet, fakeDocker := newTestKubelet(t) fakeDocker.Images = []docker.APIImages{ { ID: "foo", }, { ID: "bar", }, } if err := kubelet.GarbageCollectImages(); err != nil { t.Errorf("unexpected error: %v", err) } if len(fakeDocker.RemovedImages) != 2 || !fakeDocker.RemovedImages.Has("foo") || !fakeDocker.RemovedImages.Has("bar") { t.Errorf("unexpected images removed: %v", fakeDocker.RemovedImages) } } </del> func TestParseResolvConf(t *testing.T) { testCases := []struct { data        string

I've been struggling on a problem over the past week, that a few pods always fail to run. When I tried to manually run one of those, I got: I could succesfully run the image on another os with different docker version and different backend filesystem, so I did many tests, to conclude that the problem is neither with btrfs, nor with docker, nor with coreos. I could create another aws autoscaling group, reusing the same versions of those tools, but without kubernetes, and I could sucessfully run that image there as well. I then removed all components from my cloudformation template, and readded them progressively, to notice that pulling this image only fails when the kubelet is running. I then tried to run the kubelet with different options, but the bare minimum is enough for docker to fail on that image. Looking at the traceback: Does kubelet try to query when only and are available? I still don't understand what does kubelet has to do with the btrfs error raised by docker, but the evidence is that everytime I stop kubelet, docker has no problem with the image: This issue is a blocker. Since I cannot get the cluster-side logging to work yet (), I'm trying to have a temporary fluentd pod running for application-side logging at first. Any suggestion?
Could this and be related? If so, think this should be p0. Who feels best equipped to dig on this one? , Adrien Lemaire wrote:
Yes, I believe it's the same issue :)
Maybe not related, but CoreOS moved from btrfs to overlay, see kubernetes issues -- sorry I don't have a CoreOS issue.
is most likely person to be able to know more about this issue.
If it happens again, it could help if someone could grab for all kubelet and dockerd threads (). That would allow us to determine if anyone is currently blocked in the kernel in a devicemapper-related system call.
Which version of kubernetes / kubelet are you running?
Ok I could re-produce the issue on aufs from HEAD: when kubelet is running: After I stopped kubelet process, every runs fine. I am going to mark this as P0, and assign it to myself.
Ok, I think I know the root cause of the issue here. Actually it is indeed caused by Image GC collection () we introduced recently.

Current support of gc of image depending on docker's "dangling=true", which gives us a wrong set unused images on a node. Instead it only returns a list of image layers without tag. When a image is pulling, all layers are untagged initially, then tagged later. In this case, some intermediate layers might be removed by kubelet GC, especially with a big image. We fixed the similar issues by adding lock during docker image pull triggered by kubernetes. But there is protection for docker image pull outside kubelet. To prove my hyperthesis, I add some debugging line to Kubelet, then run From below, one can see the same image layer is being removed. I did reopen to re-do Image GC, but didn't treat it as the highest issue in my queue. Now with these issues being reported, I think our current image GC does nothing except introducing issues, and we should remove it or completely disable it: 1) Issues reported like this. And it is very hard, or unnecessary complicated to fix this issue. 2) I think when we decided if a image should be GC, it should deal the entire image stacks, not one or two image layers. Even we might fix all race issues today, but we might break the logic later. Removing one or two intermediate layers is very hard to debug. 3) Since both docker and kubernetes pro all images having tag now. Any images with proper tag wouldn't be removed at all even they are not referred by any container for long time. cc/
is trying to disable garbage collection temporarily.
thank you !
I got this error on CoreOS, current stable (at this date) on DigitalOcean, for a single container that I had never built before -- rebuilding it after changing a single file making up the image, made the problem go away.


Question: "k8s.io/apimachinery/pkg/util/wait" k8sclientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" <del> openapiutil "k8s.io/kube-openapi/pkg/util" </del> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/utils/crd" <del> "sigs.k8s.io/yaml" </del> ) var (

Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] Since when has it been failing: 04-26 05:02 PDT Testgrid link: Reason for failure: Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: stderr: the server doesn't have a resource type "ksvc" error: exit status 1 Full Stack Trace () +0x1545 (0xc002ca6100) +0x324 (0xc002ca6100) +0x2b testing.tRunner(0xc002ca6100, 0x4b8d060) +0xc9 created by testing.(T).Run +0x350 [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. Anything else we need to know: /cc /cc /milestone v1.19 /priority critical-urgent
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

<detailsIn response to : Which jobs are failing: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for CRD with validation schema [Conformance] 04-26 05:02 PDT Apr 26 12:28:01.219: FAIL: failed to explain error running /workspace/kubernetes/platforms/linux/amd64/kubectl --server=https://35.185.126.243 -- explain Command stdout: the server doesn't have a resource type "ksvc" exit status 1 () (0xc002ca6100) (0xc002ca6100) testing.tRunner(0xc002ca6100, 0x4b8d060) created by testing.(T).Run [AfterEach] [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] ï¿½[1mSTEPï¿½[0m: Collecting events from namespace "crd-publish-openapi-4377". ï¿½[1mSTEPï¿½[0m: Found 0 events. Apr 26 12:28:01.314: INFO: POD NODE PHASE GRACE CONDITIONS Apr 26 12:28:01.314: INFO: Apr 26 12:28:02.899: INFO: skipping dumping cluster info - cluster too large Apr 26 12:28:02.899: INFO: Waiting up to 3m0s for all (but 50) nodes to be ready ï¿½[1mSTEPï¿½[0m: Destroying namespace "crd-publish-openapi-4377" for this suite. /cc /milestone v1.19 </details
/milestone v1.19 Thanks for opening this! I'll get you on milestone maintainers soon :) Also, was taking a look at this as well and noticed this test was recently :
As noted in , this modified an existing conformance test and made it flaky. Please move the new test out of the conformance test and mark it as flaky until this issue has been resolved.

Cc for flake introduced in
It is a bit weird that I cannot reproduce this problem on my local kind cluster by running even after 20 times I suspect this is a cache issue since I once created a crd without short name and forgot to delete it in the test.
/retitle [Flaky Test]: gce-master-scale-correctness (ci-kubernetes-e2e-gce-scale-correctness) /kind flake /remove-kind failing-test
/assign
I can reproduce this problem by the following steps now. It looks like that this problem has something to do with discovery cache. <pre$ cat <summary<detailsapiVersion: kind: CustomResourceDefinition metadata: name: spec: conversion: strategy: None group: crd-publish-openapi-test- names: kind: Service listKind: ServiceList plural: services singular: service shortNames: ["ksvc"] scope: Namespaced versions: name: v1alpha1 schema: openAPIV3Schema: properties: spec: description: Specification of CustomService properties: dummy: description: Dummy property. type: string type: object type: object served: true storage: true </details</summary$ kubectl get crd No resources found in default namespace. $ rm -r $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl create -f created $ kubectl explain the server doesn't have a resource type "ksvc" $ kubectl explain <summary<detailsKIND: Service VERSION: crd-publish-openapi-test- RESOURCE: spec <ObjectDESCRIPTION: Specification of CustomService FIELDS: dummy <stringDummy property. </details</summary</pre
looks like
/assign


Question: Limits: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(nodeMaxAllocatable*5/10, "DecimalSI"), }, <ins> Requests: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(nodeMaxAllocatable*5/10, "DecimalSI"), }, </ins> }, } WaitForSchedulerAfterAction(f, createPausePodAction(f, conf), ns, podName, false)

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: // Expects the OpenStack Cinder CSI Driver to be installed and configured on all nodes. CSIMigrationOpenStackComplete featuregate.Feature = "CSIMigrationOpenStackComplete" <del> // owner: @MrHohn // alpha: v1.15 // beta:  v1.16 // GA: v1.17 // // Enables Finalizer Protection for Service LoadBalancers. ServiceLoadBalancerFinalizer featuregate.Feature = "ServiceLoadBalancerFinalizer" </del> // owner: @RobertKrawitz // alpha: v1.15 //

When should it be removed? /sig network /kind cleanup
It was GAed in 1.17 and should have been removed in 1.18. I will send a PR to remove it retrospectively. /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I know a new-contributor looking for good-first-issues. Do you mind if I cc him on this issue since it seems pretty straightforward ?
That sounds perfect, thanks!
cc
I am still planning on working on this, I am just having some dev environment issues I am hoping to resolve in the next few days. I'll update again when I get them resolved.
Hey, just a quick update. I have been fairly busy and am going on vacation next week. I AM still planning on working on this, however since I am obviously very slow if someone comes along and wants to work on this feel free. Just please comment that you will work on it so I know :).


Question: Containers:                    p.ContainerSpecs(), }, } <ins> if framework.NodeOSDistroIs("windows") { thePod.Spec.NodeSelector = map[string]string{ "kubernetes.io/os": "windows", } } return thePod </ins> } // QualifiedServiceAddress returns the address that can be used to hit a service from

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake"

No response No response_ /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cloud-provider
Fun. Are these test executed with special arguments or similar when running in CI? This is locally on master for me on my laptop:
This is currently blocking 1.29 beta.0 release cut. Do you think this is a release blocker? If yes, then can you please give an estimated time for fix? Thanks!
No, I don't think this is a release blocker. The problem is the test, not the implementation. The PR I have up should de-flake the test and fix the issue.
Thanks a lot for confirming
This is now the top flaking test in " service": 11,
ping
The PR fixing this is still not approved by the corresponding SIG, yet (ref ).
/assign This controller is really more sig-network than cloud provider :)
we discussed this at the sig cloud provider meeting today, but will need more time to investigate.
seeing this frequently in and in downstream CI unit test runs
should we revert if we cannot fix the test that came in with it? are we sure that PR didn't introduce a bug?
unit tests must not be flaky ... we either need to prioritize fixing of the flake (either the open PR or an alternative), or back out the change that brought in the flaky test. looks low risk to me and consistent with needing to wait for an informer to sync before the test will be deterministic


Question: staticResources:         staticResources, kubeClient:              kubeClient, kubeletClient:           kubeletClient, <ins> recorder:                recorder, </ins> registerRetryCount:      registerRetryCount, podEvictionTimeout:      podEvictionTimeout, deletingPodsRateLimiter: deletingPodsRateLimiter,

/cc
Is this from head? I've never seen this before, maybe changed anything recently. Anyway, the code in question will be deleted by
I saw this in the Travis run for an unrelated PR (Kubelet test refactoring). If it's going away, then that's ho we'll fix it :)
Yup. It'll go away. I'll assign it to myself - I'll run test few times after to make sure it's gone.
another example: https://travis-
Just hit this again: https://travis-
I was hoping we could finish getting rid of this code this week, which is why I had been ignoring this. But we ran out of time and Poland has a holiday on Monday, so I guess we need to fix this race. I'll take a look.
Thanks. This is becoming a real nuisance that is masking other failures.
To repro I assume I just type the commands from Travis, e.g. the shell stuff in https://travis- ?
Never mind, I got it. Though the first time I ran it all the tests passed.
I don't think the failure is deterministic, but yes.
This is super annoying, it happens way too much of the time...
Sigh, I was hoping we can get rid of the code this week. I think the problem comes from someone changes the event recording. I'll send a PR
It seems to be basically always failing now, except when some other flake fails first.
It would be fine with me if we just temporarily disabled event generation there.
If I'm reading the stack traces correctly, what's going on is that NewNodeController [] calls StartRecordingToSink() [] which calls StartEventWatcher() which calls eventHandler() which calls StartRecordingToSink() which calls sink.Create() which is Create() in , which appends to c.Fake.Actions. At the same time, the last line of the test tries to print "item.fakeNodeHandler.Actions" and these are presumably the same variable. I'm not sure why these are running at the same time, though.
+1 to Braians idea to temporary disable event generation.
Re-titled issue.
Re-titled issue.


Question: {utiliptables.TableFilter, KubeForwardChain}, } <ins> var iptablesCleanupChains = []struct { table utiliptables.Table chain utiliptables.Chain }{ {utiliptables.TableNAT, kubeServicesChain}, {utiliptables.TableNAT, kubePostroutingChain}, {utiliptables.TableNAT, KubeFireWallChain}, {utiliptables.TableNAT, KubeNodePortChain}, {utiliptables.TableNAT, KubeLoadBalancerChain}, {utiliptables.TableFilter, KubeForwardChain}, } </ins> // ipsetInfo is all ipset we needed in ipvs proxier var ipsetInfo = []struct { name    string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig network What happened: Kube-proxy () deployed as daemonset with ipvs mode produce during startup following error: With verbosity -v5 the logs looks like this: Looks like proxier is trying to perform cleanup part: but failed. Such error is generated only once during startup. What you expected to happen: Error should not exist :) Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: none OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Install tools: custom script based on docker Others:
I've build kube-proxy with additional information: and the logs looks like this:
same issue
Looks like also fixed this one.
I'm on 1.12.0 and the issue is still there I suppose is included in 1.12?
Updated my kube to 1.12.3 and I got same
I'm still seeing this on v1.12.4 as well
same here on v1.13.2, throws this error when kube-proxy initializes. Running on CoreOS in ipvs mode:
Hello, I also encountered this problem in Kubernetes 1.13.3. The operating system is CentOS 7.6. The kernel version is: Linux linux-node4 3.10.0-957.el7.x86_64
Why is this closed? It looks like an unresolved issue. I have the same errors with 1.12.1.
same as me
same as me,1.12.3
same problem OS: Ubuntu 18 kernel: 4.15.0-46-generic Kubernetes: 1.13.4
same problem OS: CentOS 7.6 kernel: 3.10.0-957.el7.x86_64 Kubernetes: 1.13.4
I met this issue on Centos 7.6 with NetworkManager installed, it seems related to the network.
Same problem OS: CentOS 7.4 Kubernetes: 1.12.5 NetworkManager not installed.

Some of the nodes does not have this issue while others do
I am getting in 1.14.0 as well in some worker nodes.
same problem,how to solve it
same problem 1.13.2 ,how fix it?
i update kube-proxy 1.14.2, the error message is disappear
same problem OS: Debian 8 Kubernetes: v1.15.0 iptables v1.4.21 kube-proxy mode ipvs I write some test code like kube-proxy --cleanup and found line 7 is , and run , the error is The problem is we should flush KUBE-LOAD-BALANCER and KUBE-NODE-PORT before delete KUBE-MARK-MASQ , I can try fix it.


Question: csiDriverRegistrarImage     string = "quay.io/k8scsi/driver-registrar:v0.2.0" ) <del> func externalAttacherServiceAccount( </del> <ins> func csiServiceAccount( </ins> client clientset.Interface, config framework.VolumeTestConfig, teardown bool,

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: func deletePDWithRetry(diskName string) { var err error <del> for start := time.Now(); time.Since(start) < 180*time.Second; time.Sleep(5 * time.Second) { </del> <ins> for start := time.Now(); time.Since(start) < gcePDRetryTimeout; time.Sleep(gcePDRetryPollTime) { </ins> if err = deletePD(diskName); err != nil { framework.Logf("Couldn't delete PD %q. Sleeping 5 seconds (%v)", diskName, err) continue

For: "Pod Disks should schedule a pod w/ a RW PD, remove it, then schedule it on another host" "Pod Disks should schedule a pod w/ a readonly PD on two hosts, then remove both." (this is not happening on v1.3 master with v1.1 nodes, nor with v1.2 tests against a v1.3 cluster) View logs and artifacts at
Mind taking a look? You should be able to repro by turning up a v1.3 cluster and running v1.1 tests against it.
for your queue
I found the cause of the test failure. The following are the main steps in the test host0Pod, and wait for it running a file to the host0Pod, and then delete it host1Pod, and waiting for it running the file from host1Pod, and then delete it disk with deletePDWithRetry() function, which has 3 minutes timeout defer() function, clean up test environment, and also run deletePDWithRetry() again just in case The problem arises from step 5. In the above test, deletePDWithRetry tried to delete PD every 5 second, but failed in 3 minutes. After it exits retry loop, it failed for the expectNoError(err, "Error deleting PD") in this function. Although it deleted the PD successfully right after this during the clean up phase (step 6). So the main issue for this test failure is the timeout is a little bit short. To fix it issue, we can either increase the timeout, or not fail when calling deletePDWithRetry for the first time. We can fail it if it cannot delete it after calling deletePDWIthRetry second time in step 6. The following is the retry loop and error code in the deletePDWIthRetry(). In current 1.3 version, we only call deletePDWithRetry once in defer() function during clean up. I think to be safe we should increase timeout to 5 minutes, and set it in the constance instead of coding it directly inside of the function.
So two options to fix this: port test fixes from main branch to the 1.1 branch to make the test more robust against delays. for to be merged, which will decrease the time it takes for a volume to be detached and therefore how long it takes to be deleted.


Question: assert.Equal(t, testLogs[4]+compressSuffix, logs[2].Name()) assert.Equal(t, testLogs[2]+"."+timestamp, logs[3].Name()) assert.Equal(t, testLogs[3], logs[4].Name()) <del> c.queue.ShutDown() </del> } func TestClean(t *testing.T) {

TestRotateLogs from kubelet/logs/containerlogmanagertest Probably since the merge of change No response_ /sig node
/triage accepted /cc
/cc
/priority important-longterm Seems to impact on this arch only?
cc
/sig windows
ack.. Checking
what is the easiest way for me to test this one locally ? using based vm ?
poke and on slack please
unfortunately, I couldn't reproduce this flake locally. Will update here once I am able to.
I am now able to see the flake, when I run the test within a Pod with a resource limit as our CI job does. Steps to reproduce: clone kubernetes test -c -race - TestRotateLogs root stress - TestRotateLogs Please find the attached trace file
This is with or on a native x86_64 as well ?
I could reproduce this even on x8664. Also, the failures do happen even when run on the VM itself and NOT ONLY within the pod as I mentioned above. The flakiness % is higher when run from a pod rather than a VM.
I was able to reproduce this on regular machines as well with while running tests. Let me check and see what needs to be done to fix this.
This seem to take care of the issue. Do you think the poll interval of is too much ? cc
bash stress - TestRotateLogs -test.v 5s: 0 runs so far, 0 failures 10s: 48 runs so far, 0 failures 15s: 96 runs so far, 0 failures 20s: 146 runs so far, 0 failures 25s: 196 runs so far, 0 failures 30s: 244 runs so far, 0 failures 35s: 292 runs so far, 0 failures 40s: 339 runs so far, 0 failures 45s: 387 runs so far, 0 failures
Spoke too early.

bash 1m30s: 812 runs so far, 4 failures (0.49%) /var/folders/l3/tf0xj3gj67bfmk13h462668w0000gn/T/go-stress-20240223T144024- === RUN TestRotateLogs Error Trace: Error: "[- test-log-1 - test-log-2.- - test-log-3 - test-log-3.- - test-log-3.- - test-log-4]" should have 5 item(s), but has 6 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3.-" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3.- Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-4" actual : "test-log-3.-" Diff: --- Expected … 1m35s: 851 runs so far, 5 failures (0.59%)
/assign
/triage accepted
Had to add a few additional logs to identify the issue, but has a fix now. PTAL when possible.


Question: "fmt" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: // implementation of the error interface func (f *FitError) Error() string { <del> output := fmt.Sprintf("failed to find fit for pod: %v", f.Pod) </del> <ins> output := fmt.Sprintf("failed to find fit for pod, ") </ins> for node, predicateList := range f.FailedPredicates { output = output + fmt.Sprintf("Node %s: %s", node, strings.Join(predicateList.List(), ",")) }

If a pod is not scheduled because it cannot find a fit, you run into really ugly event output like the following: This makes the output of horrific to work with. A better event would just read: The pod is known from the involved object already so no need to encode it all.


Question: InformerFactory:  informers.NewSharedInformerFactoryWithOptions(testClientset, time.Duration(1)), InformersStarted: make(chan struct{}), } <del> _, _, err := startResourceQuotaController(ctx) </del> <ins> for funcName, controllerInit := range controllerInitFuncMap { _, _, err := controllerInit(ctx) if test.expectedErr != (err != nil) { t.Errorf("%v test failed for use case: %v", funcName, name) } } _, _, err := startModifiedNamespaceController( ctx, testClientset, testClientBuilder.ConfigOrDie("namespace-controller")) </ins> if test.expectedErr != (err != nil) { <del> t.Errorf("test failed for use case: %v", name) </del> <ins> t.Errorf("Namespace Controller test failed for use case: %v", name) </ins> } } }

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: s.mux.Handle("/logs/", http.StripPrefix("/logs/", http.FileServer(http.Dir("/var/log/")))) s.mux.HandleFunc(s.prefix+"/", s.ServeREST) healthz.InstallHandler(s.mux) <del> s.mux.HandleFunc("/index.html", s.handleIndex) </del> <ins> s.mux.HandleFunc("/", s.handleIndex) </ins> // Handle both operations and operations/* with the same handler s.mux.HandleFunc(s.operationPrefix(), s.handleOperationRequest)

sig-release master-blocking build-masterci-kubernetes-build.Overall 2022-06-01 20:49 CEST, commit No response /sig release cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm checking that
/assign
this pr should fix
it looks like it's still failing: https://k8s-
/reopen reopening as this is causing all our CI to run against stale artifacts at: v1.25.0-alpha.0.734+
Reopened this issue. <detailsIn response to : /reopen v1.25.0-alpha.0.734+ </details
my previous fix only fixed one side of the things, missing now job are green ! closing for now /close
Closing this issue. <detailsIn response to : my previous fix only fixed one side of the things, missing now job are green /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank you !


Question: } // only send events newer than resourceVersion if event.ResourceVersion > resourceVersion { <del> c.sendWatchCacheEvent(event) </del> <ins> c.sendWatchCacheEvent(&event) </ins> } } }

Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.


Question: } // TODO: move this into the kubelet itself <del> func MonitorCAdvisor(k *Kubelet) { </del> <ins> func MonitorCAdvisor(k *Kubelet, cp uint) { </ins> defer util.HandleCrash() // TODO: Monitor this connection, reconnect if needed? glog.V(1).Infof("Trying to create cadvisor client.") <del> cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:4194") </del> <ins> cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:" + strconv.Itoa(int(cp))) </ins> if err != nil { glog.Errorf("Error on creating cadvisor client: %v", err) return

Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?


Question: t.Fatalf("%s: actual warning message did not match expected warning message.n Expecting:n%vn  Got:n%v", test.description, e, a) } } <ins> if len(test.expectOutputToContain) > 0 { out := outBuf.String() if !strings.Contains(out, test.expectOutputToContain) { t.Fatalf("%s: expected output to contain: %snGot:n%s", test.description, test.expectOutputToContain, out) } } </ins> }) } }

When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted


Question: }, "activeDeadlineSeconds with backofflimit reach": { 1, 1, 1, 10, 0, <del> 1, 0, 2, true, 1, 0, 0, 3, "BackoffLimitExceeded", </del> <ins> 0, 0, 1, true, 0, 0, 0, 1, "BackoffLimitExceeded", </ins> }, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: "os" "runtime" "sync" <del> "syscall" </del> "time" "unsafe" "github.com/golang/glog" cadvisorapi "github.com/google/cadvisor/info/v1" <ins> "golang.org/x/sys/windows" </ins> "k8s.io/apimachinery/pkg/util/wait" ) <ins> // MemoryStatusEx is the same as Windows structure MEMORYSTATUSEX // https://msdn.microsoft.com/en-us/library/windows/desktop/aa366770(v=vs.85).aspx type MemoryStatusEx struct { Length               uint32 MemoryLoad           uint32 TotalPhys            uint64 AvailPhys            uint64 TotalPageFile        uint64 AvailPageFile        uint64 TotalVirtual         uint64 AvailVirtual         uint64 AvailExtendedVirtual uint64 } </ins> var ( <del> modkernel32                            = syscall.NewLazyDLL("kernel32.dll") procGetPhysicallyInstalledSystemMemory = modkernel32.NewProc("GetPhysicallyInstalledSystemMemory") </del> <ins> modkernel32              = windows.NewLazySystemDLL("kernel32.dll") procGlobalMemoryStatusEx = modkernel32.NewProc("GlobalMemoryStatusEx") </ins> ) // NewPerfCounterClient creates a client using perf counters

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Windows uses GetPhysicallyInstalledSystemMemory to get physical memory installed on Windows machine and reports it as part of node metrics to heapster, etc. On VMWare Fusion, GetPhysicallyInstalledSystemMemory fails to read data from SMBIOS firmware tables and returns error code 0x0d. That causes on Windows to bail out. It is also reported on Virtualbox. What you expected to happen: Windows should be able read to physical memory on the system and shouldn't exit. How to reproduce it (as minimally and precisely as possible): Run on VMWare Fusion or Virtualbox . Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Windows Kernel (e.g. ): Install tools: Others: VMWare Fusion or Virtualbox
/sig windows /sig node
this will likely get fixed after 1.9 is released
Sure, I am fixing it. Unfortunately it missed the 1.9 train.
Hi, any idea on when this might get fixed? Thanks!
It is fixed in Got test failure in pull-kubernetes-e2e-kops-aws yesterday but not before. I will look into it shortly.
thanks for the swift reply! I need to learn some new (for me) github skills to get that PR.
i just approved the PR again. we should be good to go. can you please check


Question: replicaSets := reaper.Extensions().ReplicaSets(namespace) rsReaper, _ := ReaperFor(extensions.Kind("ReplicaSet"), reaper) <del> deployment, err := deployments.Get(name) if err != nil { return err } // set deployment's history and scale to 0 // TODO replace with patch when available: https://github.com/kubernetes/kubernetes/issues/20527 zero := 0 deployment.Spec.RevisionHistoryLimit = &zero deployment.Spec.Replicas = 0 // TODO: un-pausing should not be necessary, remove when this is fixed: // https://github.com/kubernetes/kubernetes/issues/20966 // Instead deployment should be Paused at this point and not at next TODO. deployment.Spec.Paused = false deployment, err = deployments.Update(deployment) </del> <ins> deployment, err := reaper.updateDeploymentWithRetries(namespace, name, func(d *extensions.Deployment) { // set deployment's history and scale to 0 // TODO replace with patch when available: https://github.com/kubernetes/kubernetes/issues/20527 zero := 0 d.Spec.RevisionHistoryLimit = &zero d.Spec.Replicas = 0 // TODO: un-pausing should not be necessary, remove when this is fixed: // https://github.com/kubernetes/kubernetes/issues/20966 // Instead deployment should be Paused at this point and not at next TODO. d.Spec.Paused = false }) </ins> if err != nil { return err }

It looks that we lack retry in the test. cc
As can be seen by number of references, this seems to be happening often. Bumping to P0.
is waiting to be merged.

http://pr- cc
This can be reproduced by running cc
Update: It didn't always generate errors; when this happened, the error came from
The conflict happens because when we stop the deployment, we first update it: cc


Question: sysPath := "/sys/class/iscsi_host" hostDirs, err := io.ReadDir(sysPath) if err != nil { <ins> if os.IsNotExist(err) { return portalHostMap, nil } </ins> return nil, err } for _, hostDir := range hostDirs {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: i use the standalone-cinder-provisioner and the backend storage is iscsi,the pv and volume can dynamic created,but the device can not attached the k8s minion the standalone-cinder-provisioner pod yaml use the web and the pod running: kube-system standalone-cinder-provisioner--5x6x6 1/1 Running 7 5d5h [root kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc--3140-11e9-bde3- 1Gi RWO Delete Bound kube-system/keystone-sc-pvc standard 6d4h the pv infomation: [root kubectl get pv pvc--3140-11e9-bde3- -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: cinderVolumeId: -0bc7-4e8a-879d- standaloneCinderProvisionerIdentity: creationTimestamp: "2019-02-15T16:38:33Z" finalizers: external- name: pvc--3140-11e9-bde3- resourceVersion: "" selfLink: /api/v1/persistentvolumes/pvc--3140-11e9-bde3- uid: -3140-11e9-bde3- spec: accessModes: ReadWriteOnce capacity: storage: 1Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: keystone-sc-pvc namespace: kube-system resourceVersion: "" uid: -3140-11e9-bde3- iscsi: initiatorName: iqn.2018- iqn: iqn.2099- iscsiInterface: default lun: 1 targetPortal: 19.19.19.

252:3260 persistentVolumeReclaimPolicy: Delete storageClassName: standard volumeMode: Filesystem status: phase: Bound [root standalone]# kubectl get po --all-namespaces -w NAMESPACE NAME READY STATUS RESTARTS AGE default iscsi-standalone 0/1 ContainerCreating 0 2s [root standalone]# kubectl describe pod iscsi-standalone Name: iscsi-standalone Namespace: default Priority: 0 PriorityClassName: <noneNode: 12.12.12.30/12.12.12.30 Start Time: Tue, 26 Feb 2019 17:15:19 +0800 Labels: <noneAnnotations: <noneStatus: Pending IP: Containers: iscsi: Container ID: Image: Image ID: Port: <noneHost Port: <noneState: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <noneMounts: from default-token-frzl8 (ro) /var/www/html from mypd (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: mypd: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: keystone-sc-pvc-storage13 ReadOnly: false default-token-frzl8: Type: Secret (a volume populated by a Secret) SecretName: default-token-frzl8 Optional: false QoS Class: BestEffort Node-Selectors: <noneTolerations: for 300s for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m48s default-scheduler Successfully assigned default/iscsi-standalone to 12.12.12.30 Normal SuccessfulAttachVolume 4m48s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc--39a5-11e9-b3a9-" Warning FailedMount 30s (x10 over 4m40s) kubelet, 12.12.12.30 MountVolume.WaitForAttach failed for volume "pvc--39a5-11e9-b3a9-" : open /sys/class/iscsihost: no such file or directory Warning FailedMount 29s (x2 over 2m45s) kubelet, 12.12.12.

30 Unable to mount volumes for pod "iscsi-standalone_default(-39a7-11e9-bb03-)": timeout expired waiting for volumes to attach or mount for pod "default"/"iscsi-standalone". list of unmounted volumes=[mypd]. list of unattached volumes=[mypd default-token-frzl8] What you expected to happen: the pod can running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:k8s v1.13 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
the k8s not connect the iscsi,so the /sys/class/iscsihost dir not exist,then attachdisk function : // Build a map of SCSI hosts for each target portal. We will need this to // issue the bus rescans. portalHostMap, err := b.deviceUtil.GetISCSIPortalHostMapForTarget(b.Iqn) if err != nil { return "", err } the GetISCSIPortalHostMapForTarget will return err then attach failed,why set the func to the login above?or the /sys/class/iscsihost dir when exist or how can i let the dir appear before run the GetISCSIPortalHostMapForTarget function? thanks
/sig storage
/assign bswartz
/assign
can you review this issue ?thanks
I see a bug here. If the iscsi modules haven't even been loaded yet, the /sys/class/iscsi_host directory won't exist. In this case it's better to return an empty map rather than an error.
yes, if the iscsi can not Initializationthe dir will not exist,maybe return empty map rather than an error or delete the func GetISCSIPortalHostMapForTarget i have commit one pr to resove the problem,can you review it?thanks


Question: "github.com/spf13/cobra" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/meta" <ins> "k8s.io/kubernetes/pkg/api/unversioned" batchv1 "k8s.io/kubernetes/pkg/apis/batch/v1" "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" </ins> client "k8s.io/kubernetes/pkg/client/unversioned" "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util"

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: #  TIMEOUT: For REMOTE=true only. How long (in golang duration format) to wait #    for ginkgo tests to complete. Defaults to 45m. #  PARALLELISM: The number of ginkgo nodes to run.  Defaults to 8. <del> #  RUNTIME: Container runtime to use (eg. docker, remote). #    Defaults to "docker". </del> #  CONTAINER_RUNTIME_ENDPOINT: remote container endpoint to connect to. <del> #    Used when RUNTIME is set to "remote". </del> <ins> #    Defaults to "/run/containerd/containerd.sock". </ins> #  IMAGE_SERVICE_ENDPOINT: remote image endpoint to connect to, to prepull images. #    Used when RUNTIME is set to "remote". #  IMAGE_CONFIG_FILE: path to a file containing image configuration.

$ make test-e2e-node fails with an error because of the docker deprecation. I think we have to fix the default options for helper scripts. $ make test-e2e-node succeeds in common cases by default. $ make test-e2e-node No response_ <details</details<details</details<details</details<details</details<details</details<details</details/assign
/sig node
/cc
/assign
/triage accepted /assign /priority important-soon