Question: return j } <ins> func newJob(parallelism, completions, backoffLimit int32, completionMode batch.CompletionMode) *batch.Job { return newJobWithName("foobar", parallelism, completions, backoffLimit, completionMode) } </ins> func newControllerFromClient(kubeClient clientset.Interface, resyncPeriod controller.ResyncPeriodFunc) (*Controller, informers.SharedInformerFactory) { sharedInformers := informers.NewSharedInformerFactory(kubeClient, resyncPeriod()) jm := NewController(sharedInformers.Core().V1().Pods(), sharedInformers.Batch().V1().Jobs(), kubeClient)

Prior to 1.24.0 setting in a Job object () would kill the Job and Pods if they ran longer than the integer specified. I tested today with 1.24.0 and it's not working. The Job defined below (for testing) should be killed 10 seconds, but isn't. It's actually being terminated after about 55 seconds. The Job to be killed after 10 seconds as per . It's worked like this in 1.23.x. On a cluster running 1.24.0 with the following YAML file called . Create the Job. On Kubernetes clusters earlier than 1.24.0 it would fail after 10 seconds and the following command should show in the events 10 seconds after the Pod is created. In K8s 1.24.0 it's terminated after about a minute instead of 10 seconds. It works as expected in 1.23.6 and other point versions of 1.23. It works as expected, and documented, in all versions of 1.23 I've tested on. Docs: <details</details<detailsAll and also local clusters. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig apps
I just searched for at locally and could not see any logic that compares running time with and marks a pod for termination. Do I need to look somewhere else?
It's here Here's the quote: "Another way to terminate a Job is by setting an active deadline. Do this by setting the field of the Job to a number of seconds. The applies to the duration of the job, no matter how many Pods are created. Once a Job reaches , all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded." I may be misunderstanding it, but I don't think I am as it works the way described in older versions. The Job owns the Pod, you set on the Job to a value, let's say . This will cause the Job to be killed if it's still running after 10 seconds. As the Job owns the Pod(s) it cleans them up as well. This is exactly how it works in previous versions.

The job controller for sure has the code for cleanup on active deadline still in place +1 from me as well. The behavior for sure is not the same on 1.23.x vs 1.24.0 using the same job manifest. And the documentation v/s the behavior is not the same.
Seem to be a side effect of feature gate. Disabling that gives me the old behavior as in 1.23.x You can get the same behavior even in 1.23.x with the feature flag enabled.
The weird bit as you see is the pod actually completes but when it processes the completion event it notices the activeDeadline value and marks deadline Exceeded while the pod actually ran to completion
/assign
I think I have figured out why this happened. It is a side effect of feature gate (thanks ). : when we call with multiple times, the workqueue will only keep the duration that will be triggered nearest from now. (when there is no feature gate or do not enable it): job-controller mainly relies on the first time sync to add the job after its : (unless the field has been changed then it will be add after ). : the feature gate makes to be and it will be add after for the job when a pod of it has been updated. Then the duration which has been after in the first time sync will be overwrite to . And after the next syncJob, there will be no one to addAfter the job again. WDYT
I have came to the same conclusion as you are based on what is in the and . So, I did a dirty prototype of tracking the activeDeadline in an entirely different to avoid conflicts and it seem to be working well. If this is an acceptable change to and I would be more than happy to raise a Draft PR for review along with some additional UT to cover the active deadline cases
Emm... Usually it is not suggested to have multiple active queues in one controller, which will cause an object to be handled by multiple workers at the same time and also make the progress more complicated. My suggestion is do at the end of syncJob every time an unfinished job is reconciled, the duration should be . This is clear for everyone to understand, brings less change and no extra burden on the controller.
I was too trigger happy and did not think about the cases you mentioned above.

Thanks for the suggestion of the approach I will took a stab at your suggestion and here is a draft of the changes to get your suggestion working. I did a quick test and that seem to work fine. There were one too many termination point in call and the has no access to the object of the key. So, instead of reading it took a deferred approach as we already had named return parameters in place on the
Other approach is to use the and fetch the job via key if the is not set to This will do an additional fetch but much simpler in change
This is essentially the same as . The solution is actually very simple, but I didn't have time to implement it Can someone take it? /priority critical-urgent /triage accepted
/assign
Let me open a PR. Do you want to just add the item back to queue or retain the check for that I have in the above ?
just add it back to the queue every time there is a job sync. It's a noop if the item is already there.


Question: Containers: []v1.Container{ { Name:    "pv-recycler", <del> Image:   "gcr.io/google_containers/busybox", </del> <ins> Image:   "busybox:latest", </ins> Command: []string{"/bin/sh"}, Args:    []string{"-c", "test -e /scrub && rm -rf /scrub/..?* /scrub/.[!.]* /scrub/*  && test -z "$(ls -A /scrub)" || exit 1"}, VolumeMounts: []v1.VolumeMount{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Use the above two files create one PV and PVC, then delete the PVC, the PV will become failed but not released. The busybox recycler never succeed for power. /sig storage What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug The solution is like image, we should enable end user can customize the recycle image. It will not only fail on power but also s390x, so we should make sure the recycle image can be customized. /cc any comments for the solution?
Dup of . This issue can be closed, since we've already got flags and for this.
Just search and , but found that they are only defined as flags and no one is using them, can you please help double check, am I missing anything?
Also the claiming that it does not support multi node cluster, this is not acceptable.
: please see Please try them out, if they do not work then we can see how to get those fixed (and avoid adding yet another config option)
Yes, since we may only got only one (maybe setup several for LB, but they can be considered identical), it is necessary for running on non-amd64 platorm to have the right busybox image, which is exactly trying to fix. I've reopened . , so the simplest way is just replacing to . And docker will help us choose the right os-arch busybox image. What's remaining that mentioned is whether we should adding a new flag to add registry url for such recycler pod image exclusively.
Thanks and I also found an example here We do not need the registry url as the pod template was customized by the end user and they can define where got the recycler images. The problem is that the customized recycler will not work in mixed environments as only one recycle pod can be specified in controller manager, so +1 to , we should make support multi arch.

not quite catch your idea of how to support, can you please provide some dummy yaml templates and some configuration files to explain? Thanks!
Actually all the , including , has been updated along with . This means will help us make the decision to choose the right busybox image. You can have a try to on different architectures, like and . A can help you and multi-arch docker images. With this support, on different os/arch can pull the right image with a same busybox image. As long as we have used such multi-arch images, those flags can work in a multi-node cluster.


Question: // Value of Filesystem is implied when not included in claim spec. // +optional VolumeMode *PersistentVolumeMode `json:"volumeMode,omitempty" protobuf:"bytes,6,opt,name=volumeMode,casttype=PersistentVolumeMode"` <del> // This field requires the VolumeSnapshotDataSource alpha feature gate to be // enabled and currently VolumeSnapshot is the only supported data source. // If the provisioner can support VolumeSnapshot data source, it will create // a new volume and data will be restored to the volume at the same time. // If the provisioner does not support VolumeSnapshot data source, volume will </del> <ins> // This field can be used to specify either: // * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot - Beta) // * An existing PVC (PersistentVolumeClaim) // In order to use VolumeSnapshot object types, the appropriate feature gate // must be enabled (VolumeSnapshotDataSource) // If the provisioner can support the specified data source, it will create // a new volume based on the contents of the specified PVC or Snapshot. // If the provisioner does not support the specified data source, the volume will </ins> // not be created and the failure will be reported as an event. // In the future, we plan to support more data source types and the behavior // of the provisioner may change.

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <del> func parseLosetupOutputForDevice(output []byte, path string) (string, error) { if len(output) == 0 { return "", errors.New(ErrDeviceNotFound) } </del> <ins> // getLoopDeviceFromSysfs finds the backing file for a loop // device from sysfs via "/sys/block/loop*/loop/backing_file". func getLoopDeviceFromSysfs(path string) (string, error) { // If the file is a symlink. </ins> realPath, err := filepath.EvalSymlinks(path) if err != nil { return "", fmt.Errorf("failed to evaluate path %s: %s", path, err) } <del> // losetup -j {path} returns device in the format: // /dev/loop1: [0073]:148662 ({path}) // /dev/loop2: [0073]:148662 (/dev/sdX) // // losetup -j shows all the loop device for the same device that has the same // major/minor number, by resolving symlink and matching major/minor number. // Therefore, there will be other path than {path} in output, as shown in above output. s := string(output) // Find the line that exact matches to the path, or "({path})" var matched string scanner := bufio.NewScanner(strings.NewReader(s)) for scanner.Scan() { // losetup output has symlinks expanded if strings.HasSuffix(scanner.Text(), "("+realPath+")") { matched = scanner.Text() break </del> <ins> devices, err := filepath.Glob("/sys/block/loop*") if err != nil { return "", fmt.Errorf("failed to list loop devices in sysfs: %s", err) } for _, device := range devices { backingFile := fmt.Sprintf("%s/loop/backing_file", device) // The contents of this file is the absolute path of "path". data, err := ioutil.ReadFile(backingFile) if err != nil { continue </ins> } <del> // Just in case losetup changes, check for the original path too if strings.HasSuffix(scanner.Text(), "("+path+")") { matched = scanner.Text() break </del> <ins> // Return the first match. backingFilePath := strings.TrimSpace(string(data)) if backingFilePath == path || backingFilePath == realPath { return fmt.Sprintf("/dev/%s", filepath.Base(device)), nil </ins> } } <del> if len(matched) == 0 { return "", errors.New(ErrDeviceNotFound) } s = matched // Get device name, or the 0th field of the output separated with ":". // We don't need 1st field or later to be splitted, so passing 2 to SplitN. device := strings.TrimSpace(strings.SplitN(s, ":", 2)[0]) if len(device) == 0 { return "", errors.New(ErrDeviceNotFound) } return device, nil </del> <ins> return "", errors.New(ErrDeviceNotFound) </ins> } // FindGlobalMapPathUUIDFromPod finds {pod uuid} bind mount under globalMapPath

I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details


Question: // enforcePodContainerConstraints checks for required resources that are not set on this container and // adds them to missingSet. <del> func enforcePodContainerConstraints(container *corev1.Container, requiredSet, missingSet sets.String) { </del> <ins> func enforcePodContainerConstraints(container *corev1.Container, requiredSet sets.String, missingSetResourceToContainerNames map[string]sets.String) { </ins> requests := container.Resources.Requests limits := container.Resources.Limits containerUsage := podComputeUsageHelper(requests, limits) containerSet := quota.ToSet(quota.ResourceNames(containerUsage)) if !containerSet.Equal(requiredSet) { <del> difference := requiredSet.Difference(containerSet) missingSet.Insert(difference.List()...) </del> <ins> if difference := requiredSet.Difference(containerSet); difference.Len() != 0 { for _, diff := range difference.List() { if _, ok := missingSetResourceToContainerNames[diff]; !ok { missingSetResourceToContainerNames[diff] = sets.NewString(container.Name) } else { missingSetResourceToContainerNames[diff].Insert(container.Name) } } } </ins> } }

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: - getResourceAPIGroup - listStorageV1CSINode - listStorageV1VolumeAttachment <del> - patchCoreV1NamespacedPersistentVolumeClaimStatus - patchCoreV1PersistentVolumeStatus </del> - patchStorageV1CSINode - patchStorageV1VolumeAttachment - patchStorageV1VolumeAttachmentStatus <del> - readCoreV1NamespacedPersistentVolumeClaimStatus - readCoreV1PersistentVolumeStatus </del> - readStorageV1CSINode - readStorageV1VolumeAttachment - readStorageV1VolumeAttachmentStatus <del> - replaceCoreV1NamespacedPersistentVolumeClaimStatus - replaceCoreV1PersistentVolumeStatus </del> - replaceStorageV1CSINode - replaceStorageV1VolumeAttachment - replaceStorageV1VolumeAttachmentStatus

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectDelete: false, }, { <del> description:  "DS-managed terminated pod", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{dsTerminatedPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "DS-managed terminated pod", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{dsTerminatedPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { description:  "orphaned DS-managed pod",

When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted


Question: } }) <del> By("not removing non-exist images [Conformance]", func() { </del> <ins> By("not removing non-exist images", func() { </ins> for _, conformImage := range conformImages { err := conformImage.Remove() Expect(err).Should(HaveOccurred())

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: import ( "context" <ins> "flag" </ins> "fmt" "net/http" "os"

What happened: panic: failed to find flag in global flagset (flag): cloud-provider-gce-lb-src-cidrs goroutine 1 [running]: (0xc0000ba900, 0x17df195, 0x1f) +0x1a9 (0xc00008e150) +0x438 () /home/neil/Dev/brightbox-cloud-controller- +0x90 What you expected to happen: The controller should run and stop with a '--cloud-provider cannot be empty' message How to reproduce it (as minimally and precisely as possible): Follow the out of tree instructions here Comment out the "" import line and build the cloud-controller-manager Anything else we need to know?: This worked fine prior to 1.14.0 Environment: Kubernetes version (use ): lient Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux bionic-vm 4.15.0-47-generic -Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x8664 x8664 x8664 GNU/Linu Install tools: kubeadm Others:
/sig cloud-provider
any updates?
we chatted about this the other day on Slack.
The GCE flag seems to be registered , with the panic eventually occurring .
Given some pointers on how to resolve the issue best, I'm happy to take a stab at a PR.
A workaround for external cloud providers seems to be to add a bogus version of the parameter before the call to , like this:
would it be better to do something like this in master? (and get it back ported to 1.14 branch?)

I think that should work, yeah. What I am wondering though is whether this is the way we want to go with long-term. It feels odd to me to have a specific GCE-only parameter be part of the generic bootstrapping process which we then need to handle in a special way. If it's about getting a fix out ASAP (and think about a possibly better design later), I'm all on-board with your patch. :+1:
the right thing to do is to get the sig-gcp folks to refactor that flag properly (and/or) remove the gce provider entirely (as a out of tree provider over time).
let's stop the bleeding where external providers have to make unnecessary changes like you mentioned (fake flag). can you please file a PR with the patch i proposed?
will do, thanks!
Thanks for reporting this issue and thanks for following-up Agreed with everything said. I can't recall why that flag was in there in the first place, I think it was required to get the reference CCM implementation working with GCE. Apologies that it broke other implementations. The proposed changes makes sense to me as well. We have some work coming up (likely in the next release cycle) to make more easily consumable as an external library which may involve moving into it's own repo. Let me know if anyone is interested in picking this up!
Submitted a fix at
in general, I'd be interested in helping on the move of . However, I likely won't have many cycles available until post-KubeCon EU. If the effort is low-priority and/or can wait until end of May / June though, I'd be happy to help.
I don't think we'll get to this until the v1.16 release cycle so I will loop you in then :)
oh please do then, that's a convenient timeline for me.
Sounds great Here's the backlog tracking issue


Question: // NodeTree is a tree-like data structure that holds node names in each zone. Zone names are // keys to "NodeTree.tree" and values of "NodeTree.tree" are arrays of node names. type NodeTree struct { <del> tree           map[string]*nodeArray // a map from zone (region-zone) to an array of nodes in the zone. zones          []string              // a list of all the zones in the tree (keys) zoneIndex      int exhaustedZones sets.String // set of zones that all of their nodes are returned by next() NumNodes       int mu             sync.RWMutex </del> <ins> tree      map[string]*nodeArray // a map from zone (region-zone) to an array of nodes in the zone. zones     []string              // a list of all the zones in the tree (keys) zoneIndex int NumNodes  int mu        sync.RWMutex </ins> } // nodeArray is a struct that has nodes that are in a zone.

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: } var eqs = equality.Semantic.Copy() <del> err := eqs.AddFunc( </del> <ins> err := eqs.AddFuncs( </ins> func(a, b metav1.ManagedFieldsEntry) bool { // Two objects' managed fields are equivalent if, ignoring timestamp, //	the objects are deeply equal.

When continuously re-applying the same object via a controller or kubectl with SSA the resourceVersion increases every time. I expected the resourceVersion to stay the same if the client always sends the same object via SSA, especially if the resulting object doesn't change (apart from resourceVersion & managedFields, but these are managed by the apiserver). Apply this CRD: <details</summary</detailsApply this CR: <details</summary</detailsGet state of the CR after first apply Apply the CR again Get state of the CR after second apply Diff =The resourceVersion and the "time" field of the managedField entry of the "test-manager" increases. The use case we have is a lot more complex than this simple scenario. The tl;dr is that we want to implement caching for our SSA calls, so that we don't have SSA calls on every reconcile of our controller if nothing actually changes. This is relatively hard to do if the resourceVersion increases after every apply. I took a closer look at the APIserver and ~ the following happens: "incoming" SSA patch is applied over the live object from etcd this leads to a change because has which means the field will be dropped Because of that change the "time" field in the managedField entry gets set to "time.Now()" later during admission the namespace field gets defaulted again (I also tried this with a mutating webhook, same result) On a high-level: client always applies the same object defaulting always results in the same object resourceVersion and managedFields[].time continuously go up I think there are many similar cases that would lead to this behavior. Basically whenever the admission chain mutates a field previously set via SSA. Notes: This only happens if there is 1s between each apply, otherwise the managedField[].time is identical and then the resourceVersion stays the same (because we then don't write to etcd) <detailsI also tried this with client-go v0.30 and kube-apiserver v1.30. Same result. </details<detailsRunning this natively on Apple Silicon M2. But can't imagine this makes a difference. Also got the same results in Docker Desktop on Mac. </details
(cc might be interesting for you)
/sig api-machinery
It looks less a bug but more of a intended behavior to me.

/cc /triage accepted
Bypassing the writes triggered by noop applies would break storage version migration (), so we're committed to supporting this behavior as a feature.
I am confused. Are you saying this is intended behavior for a NOP SSA apply to increment resourceVersion? This seems to not actually happen in all cases, only for certain objects/edge cases (see ). For example, try this: No resourceVersion is incremented above). Surprisingly I couldn't find any clear statement that SSA doesn't increment the resource version, but does say "a no-op apply is not very much more work for the API server than an extra GET" which seems impossible if resourceVersion is incremented
is probably only updating it because either the fieldManager changes or in the patch body the apiVersion is different (or both, perhaps)?
We need this in storage version migration to make sure we correctly handles the CRD schema changes. See Joe's comments here - If it helps to understand this better, we have tests to validate this.
I may have misspoke. When storage versions changes, a NOP SSA apply will trigger a resourceVersion bump. If the storage version is unchanged and an update would be a NOP for etcd, I believe we short-circuit the write (?). This might explain the configMap case.
For the reported issue: We've seen cases like this before where the time gets bumped causing a NOP to become a write. This could be improved. Note that this isn't technically a bug because the API server is allowed to bump the resourceVersion for any write (the discussion about storage version changes causing this is a good example of where this can happen in a way that the caller wouldn't expect). But I'd like to see the etcd write optimized away for cases like this.
Sounds similar to perhaps which had some investigation. FWIW as a controller author my expectation is that repeatedly applying does not repeatedly trigger resourceVersion increments. It's fine if it happens rarely (storage version change, etc) but if it happens consistently that means our own writes re-trigger reconciliation and we have an infinite loop of writes. AFAIK all the marketing around writing SSA controllers has been that you just apply what you want and don't try to do a get+diff+apply. Even if we did want to do that, it doesn't really work well currently ().

If we aren't able to rely on NOP writes (usually) being NOPs, I don't see how we are supposed to use SSA to write a controller at all?
This is a really good point. I agree.
looks like it was intended fix this sort of problem. I'm looking into why it doesn't work for this case.
doesn't work for CRs since the managed fields are inside of a unstructured object and are not typed as (for the slow path equality check in IgnoreManagedFieldsTimestampsTransformer).
is a draft proposal for a fix. Feedback welcome on how to make this fix cleaner. I'll test carefully before moving it out of draft.


Question: case deletePet: err = petManager.Delete(pet) } <del> if err != nil { </del> <ins> switch err.(type) { case errUnhealthyPet: // We are not passing this error up, but we don't increment numPets if we encounter it, // since numPets directly translates to petset.status.replicas continue case nil: continue default: </ins> it.errs = append(it.errs, err) } }

Is this a request for help? Nope What keywords did you search in Kubernetes issues before filing this one? petset replicas current desired Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: 24 cores, 64 GB RAM OS Arch Linux Kernel (e.g. ): Linux hostname 4.4.13-1-lts SMP Wed Jun 8 16:44:31 CEST 2016 x86_64 GNU/Linux Install tools: local-cluster- Others: What happened: While creating PetSet based on this tutorial: first PetSet pod was stuck in pending state due to lack of PersistentVolumes in cluster. When getting the petset info via I expected to see "CURRENT" set to 1, while the kubectl output was as follows: While clearly shows that only one pod is created: In json and yaml output, status.replicas value was also 2. What you expected to happen: I expected that CURRENT value shown by kubectl would be due to only one pod being created. How to reproduce it (as minimally and precisely as possible): Create via on empty cluster and run and . Anything else do we need to know: Nope
CURRENT is the number of pods that have been created, not that they are READY. See for example the latest api change for replica sets where we exactly for this reason. All controllers (petsets, daemonsets, deployments, etc.) should have ReadyReplicas cc:
See also
I guess it doesn't work anyway, since only one pod was created for this PetSet, but the CURRENT says that 2 were created. I will change bug description then.

Hmm, the idea is to sync every pet periodically, but only create them one at a time. The logic that counts status.replicas is wrong. This counter shouldn't be incremented in the case where we didn't update/sync or create a pet: Today we do increment it all pet creation is blocked on the first pet becoming unhealthy. Should be an easy fix if anyone wants to jump in ( fyi) otherwise I'll get to it once 1.4 is out.
I can try to fix it if that's OK. Thanks for the pointer to the source code!
Please do, co-assigned


Question: "loadBalancerSourceRanges":      "If specified and supported by the platform, this will restrict traffic through the cloud-provider load-balancer will be restricted to the specified client IPs. This field will be ignored if the cloud-provider does not support the feature." More info: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/", "externalName":                  "externalName is the external reference that discovery mechanisms will return as an alias for this service (e.g. a DNS CNAME record). No proxying will be involved.  Must be a lowercase RFC-1123 hostname (https://tools.ietf.org/html/rfc1123) and requires `type` to be "ExternalName".", "externalTrafficPolicy":         "externalTrafficPolicy describes how nodes distribute service traffic they receive on one of the Service's "externally-facing" addresses (NodePorts, ExternalIPs, and LoadBalancer IPs). If set to "Local", the proxy will configure the service in a way that assumes that external load balancers will take care of balancing the service traffic between nodes, and so each node will deliver traffic only to the node-local endpoints of the service, without masquerading the client source IP. (Traffic mistakenly sent to a node with no endpoints will be dropped.) The default value, "Cluster", uses the standard behavior of routing to all endpoints evenly (possibly modified by topology and other features). Note that traffic sent to an External IP or LoadBalancer IP from within the cluster will always get "Cluster" semantics, but clients sending to a NodePort from within the cluster may need to take traffic policy into account when picking a node.", <del> "healthCheckNodePort":           "healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type).", </del> <ins> "healthCheckNodePort":           "healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type). This field cannot be updated once set.", </ins> "publishNotReadyAddresses":      "publishNotReadyAddresses indicates that any agent which deals with endpoints for this Service should disregard any indications of ready/not-ready. The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery. The Kubernetes controllers that generate Endpoints and EndpointSlice resources for Services interpret this to mean that all endpoints are considered "ready" even if the Pods themselves are not. Agents which consume only Kubernetes generated endpoints through the Endpoints or EndpointSlice resources can safely assume this behavior.", "sessionAffinityConfig":         "sessionAffinityConfig contains the configurations of session affinity.", "ipFamilies":                    "IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service. This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail. This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service. Valid values are "IPv4" and "IPv6".  This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to "headless" services. This field will be wiped when updating a Service to type ExternalName.nnThis field may hold a maximum of two entries (dual-stack families, in either order).  These families must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.",

We are currently using v1.19 and it seems that if you have a service with set to then changing the health check node port is not supported. One way to get around this is by changing the to and then changing it back to in addition to specifying the with the desired port number. This behavior is not very well documented and the below error message is not very useful: I wonder if we could also add another case below to release the current health check port and allow changing it to the user specified value given it's valid? It's possible that after creating a someone wants to change it to a user-provided and the current way to do it is only by toggling the . This feature with the documentation would be more straightforward way to make any port changes.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
19 is EOL, last supported version is 1.22 It seems the message has changed since then, this area had important changes in latest releases that improved the API behavior However, the field still can not be updated once set, what you do switching the is wiping the value and reallocating it. It is a documentation problem then
IMO, the amount of effort it would take to make this mutable is not justified when there is a workaround. Convince me the workaround doesn't work? Overall HCNP is a buried implementation detail.
I can't speak much to the amount of effort, but I can speak a bit to the value: a user, I don't expect healthcheckNodePort to be immutable. I'm not sure if my mental model is too simplistic, but it feels like a part of the configuration I control, and I can change NodePorts, so why not the healthcheckNodePort. the work around isn't necessarily obvious. It also takes experimenting with to know that it'll even work. this in a production environment is not trivial.

If changing the healthcheckNodePort was supported then you could change the config to the state you want it to be, and your rollout orchestration tooling goes and applies it to your collection of k8s clusters. With the work around you now have to have a 2 step process. After step 1 of this process your service is using a traffic policy which might break your application (say you depend on client ip). So to avoid the downtime you'd have to build some reasonably off-pattern orchestration that doesn't just apply the configuration from the commit you used for your release, but instead two commits, or two configs, etc. Operationally this isn't actually that cheap to build, especially as this is likely to be a one-off operation and not something you do every release. Not exactly the same thing, but related to healthcheckNodePort, I've also found that specifying it explicitly vs not explicitly in the config, and switching between the two modes, has unexpected behavior as well. If you started out as having it specified, applying a config with it unspecified tries to allocate a new value, and complains that it can't change healthcheckNodePort for traffic policy local. Why not just use the already allocated value? Going from unallocated to specifying it explicitly at the same value seems to works fine.
The latter issue should be fixed in recent releases - what version are you seeing that on? I understand it would be more consistent to be mutable but there are real downstream implications to changing it, code complexity, test complexity, etc, and I don't understand why you want to change it in the first place. Humans should not be paying attention to that field. It should not even be in spec, but for historical reasons.
Same version as original post, v1.19. Glad to hear that issue is fixed in later versions! We're making steady progress updating, but not all caught up yet. I don't want to insist on this fix, as I said I'm not aware of the fix complexity and leave it up to you guys to balance that out, just wanted to provide the value perspective. As for not being exposed to users at all, I think we'd probably have been better off if we never knew about it.

Unfortunately our original configs had it explicitly set, eventually ran into conflicts in various k8s clusters and tried to transition to having it be unspecified, and thus ran into the above situation.
Yeah, there's really no good reason to set it, pretty much ever, and lots of reasons not to. I will cross-check that it is fixed if I get a moment tomorrow.
It has been fixed by you Tim
I thought so, I just wanted to triple check :)


Question: "--cluster-cidr=$(${kube_env}['CLUSTER_IP_RANGE'])", "--hostname-override=${instance_name}" ) $kubeproxy_args = ${default_kubeproxy_args} + ${kubeproxy_args} Log-Output "Final kubeproxy_args: ${kubeproxy_args}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There's a race condition between starting Windows service and when the VMs network is actually available to communicate with the VM's metadata server. If the service starts first and cannot communicate with the metadata server or start talking to the stackdriver logging service it'll crash and not resume because the service is not configured to restart on failure. Typically a is used for a service like this but Windows does not support privileged containers so a standard Windows Service is used. What you expected to happen: If the service fails to start it should be automatically be restarted. How to reproduce it (as minimally and precisely as possible): This is a flaky issue but a way to repro this is to effectively by RDP'ing into the node and running: Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 and onward Cloud provider or hardware configuration: GCE OS (e.g: ): Windows Kernel (e.g. ): Windows (any version) Install tools: cluster/gce/windows/README-GCE-Windows-kube- Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign jeremyedwards
GitHub didn't allow me to assign the following users: jeremyedwards. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign jeremyedwards Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/resource" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/strategicpatch" </ins> ) // AnnotateOptions have the data required to perform the annotate operation

and work in this pattern: first issuing a "GET" to the API server to retrieve the latest object, then adding the label/annotation to the object, and finally issuing an "PUT" with the modified object. The request will fail if the object stored in the server has been updated between the "GET" and "PUT", even if the update has no conflict with the label/annotate request. In such cases, instead of returning an error, should retry without disturbing the end user. There are two ways to implement this: in kubectl: simply retry the label/annotate process several times if the server rejected the request because the object has been updated. The only exception is when the is set, then kubectl shouldn't retry. the strategicMergePatch instead of calling Update at client-side. PATCH is more robust because PATCH does the "get, modify, update" in the server-side, so there is less chance to have a conflict, and PATCH will retry at the server-side (see ). We need to make some adaptions though: kubectl annotate/label has the , when it's false, kubectl should retrieve the latest object to confirm there is no existing annotation/label of the same key, and then proceed with issuing the patch. if the is set, [UPDATE:] we just need to specify the resourVersion in the patch, then it will be treated as a precondition by the UpdateResource() handler in the API server. I plan to take the first approach as an immediate fix.
Immediate need is to fix the flaky e2e test AIUI.
e2e? do we have an issue out there? is an immediate fix to the flake in test-
If this is for 1.1, it needs to be P0 at this point. Should it be?
Fwiw I use annotations both from cmd line as an admin and from scripts, to experiment with loadbalancer settings. In the case of the former, I've never really had a collision, and in the case of the latter my scripts know how to retry like we retry all other updates.
Thank you When used in scripts, the annotate and label commands have collisions frequently, e.g. in the test-, see . IMO, we shouldn't require users to retry in their scripts if we know there is a simple way to avoid the collisions. Anyway, and are sent to fix this.


Question: package nodelifecycle import ( <ins> "fmt" "sync" "time" "github.com/golang/glog" </ins> "k8s.io/api/core/v1" apiequality "k8s.io/apimachinery/pkg/api/equality" apierrors "k8s.io/apimachinery/pkg/api/errors"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug Description: If TaintNodesByCondition enabled, we should disable and predicates in scheduler. In TaintNodeByCondition, we will handle those conditions: , , and ; NetworkUnavailable checking is removed by predicates, but and not (no predicate for ). /cc
/sig scheduling /sig node
/assign


Question: "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", "//vendor/k8s.io/klog:go_default_library", <ins> "//vendor/k8s.io/utils/exec:go_default_library", </ins> "//vendor/k8s.io/utils/keymutex:go_default_library", "//vendor/k8s.io/utils/strings:go_default_library", ],

What happened: In order to communicate with the target, the iscsi plugin creates a new iface. If for some reason an iface with the same name already exists in the node, this command will re-create it and no error will be reported. However, if the iface already exists and is being used by a session, this command will fail and report the error code 15 (). That's because it can't re-create the iface while it's in use by a session. What you expected to happen: The plugin could go ahead and use the existing iface instead of failing. How to reproduce it (as minimally and precisely as possible): pod1 pod2 Here, will re-use the session/iface from the step above. pod1 At this point, k8s will try to delete the session/iface used by pv1. However, it'll fail because the session is currently in use by pv2. pod1 Here the pod's status will be stuck at : Templates used: Anything else we need to know?: problem happens when a custom initiator is used, and it was reported to me that it's necessary in some set-ups in order to "restrict access to LUNs of the storage". to reproduce manually (assuming there's a session using the iface): /sig storage /CC
/assign
is this related to cc
If iface is reused, wouldn't it try to delete it once last volume (from kubelet perspective) is detached?
No


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubectl import ( "reflect" "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" ) func TestGenerateService(t *testing.T) { tests := []struct { params   map[string]string expected api.Service }{ { params: map[string]string{ "selector":       "foo=bar,baz=blah", "name":           "test", "port":           "80", "protocol":       "TCP", "container-port": "1234", }, expected: api.Service{ ObjectMeta: api.ObjectMeta{ Name: "test", }, Spec: api.ServiceSpec{ Selector: map[string]string{ "foo": "bar", "baz": "blah", }, Port:          80, Protocol:      "TCP", ContainerPort: util.NewIntOrStringFromInt(1234), }, }, }, { params: map[string]string{ "selector":       "foo=bar,baz=blah", "name":           "test", "port":           "80", "protocol":       "UDP", "container-port": "foobar", }, expected: api.Service{ ObjectMeta: api.ObjectMeta{ Name: "test", }, Spec: api.ServiceSpec{ Selector: map[string]string{ "foo": "bar", "baz": "blah", }, Port:          80, Protocol:      "UDP", ContainerPort: util.NewIntOrStringFromString("foobar"), }, }, }, { params: map[string]string{ "selector":       "foo=bar,baz=blah", "name":           "test", "port":           "80", "protocol":       "UDP", "container-port": "foobar", "public-ip":      "1.2.3.4", }, expected: api.Service{ ObjectMeta: api.ObjectMeta{ Name: "test", }, Spec: api.ServiceSpec{ Selector: map[string]string{ "foo": "bar", "baz": "blah", }, Port:          80, Protocol:      "UDP", PublicIPs:     []string{"1.2.3.4"}, ContainerPort: util.NewIntOrStringFromString("foobar"), }, }, }, { params: map[string]string{ "selector":                      "foo=bar,baz=blah", "name":                          "test", "port":                          "80", "protocol":                      "UDP", "container-port":                "foobar", "public-ip":                     "1.2.3.4", "create-external-load-balancer": "true", }, expected: api.Service{ ObjectMeta: api.ObjectMeta{ Name: "test", }, Spec: api.ServiceSpec{ Selector: map[string]string{ "foo": "bar", "baz": "blah", }, Port:                       80, Protocol:                   "UDP", PublicIPs:                  []string{"1.2.3.4"}, ContainerPort:              util.NewIntOrStringFromString("foobar"), CreateExternalLoadBalancer: true, }, }, }, } generator := ServiceGenerator{} for _, test := range tests { obj, err := generator.Generate(test.params) if !reflect.DeepEqual(obj, &test.expected) { t.Errorf("expected:n%#vngotn%#vn", &test.expected, obj) } if err != nil { t.Errorf("unexpected error: %v", err) } } } </ins>

expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.


Question: - apiGroups: - "" resources: <ins> - nodes verbs: - get - list - apiGroups: - "" resources: </ins> - events verbs: - watch

Is this a request for help? : No What keywords did you search in Kubernetes issues before filing this one? : PVC, Openstack, RBAC Is this a BUG REPORT or FEATURE REQUEST? : BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.666+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-05-30T09:26:07Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-47-generic Install tools: kubeadm Others: N/A What happened: kube-controller-manager could not list nodes becase of missing permission for persistent-volume-binder service account. What you expected to happen: persistent-volume-binder service account should be able to list nodes. How to reproduce it (as minimally and precisely as possible): bootstrap a cluster with RBAC and try to create cinder PVC on openstack. Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig auth


Question: waitForNumProxyLoops(t, p, 1) stopProxyByName(p, service) <ins> if svcInfo.isAlive() { t.Fatalf("wrong value for isAlive(): expected false") } </ins> // Wait for the port to really close. if err := waitForClosedPortTCP(p, svcInfo.proxyPort); err != nil { t.Fatalf(err.Error())

By looking why sometime my sites magicaly slow down for a short period of time I got that : each 5 seconds dns query are jumping from < 2 ms to that . until [[ 0 -eq 1 ]]; do dig mysql-change | grep -E "time: [0-9]{3,}" && date; done ; Query time: 2187 msec Wed Jul 29 13:39:31 UTC 2015 ; Query time: 2114 msec Wed Jul 29 13:39:36 UTC 2015 ; Query time: 1890 msec Wed Jul 29 13:39:41 UTC 2015 ; Query time: 1874 msec Wed Jul 29 13:39:46 UTC 2015 ; Query time: 1930 msec Wed Jul 29 13:39:51 UTC 2015 ; Query time: 1942 msec Wed Jul 29 13:39:56 UTC 2015 ; Query time: 2350 msec Wed Jul 29 13:40:01 UTC 2015 ; Query time: 2469 msec Wed Jul 29 13:40:06 UTC 2015 ; Query time: 1898 msec Wed Jul 29 13:40:11 UTC 2015 ; Query time: 2083 msec Wed Jul 29 13:40:16 UTC 2015 ; Query time: 2110 msec Wed Jul 29 13:40:21 UTC 2015 ; Query time: 1921 msec Wed Jul 29 13:40:26 UTC 2015 ; Query time: 2023 msec Wed Jul 29 13:40:31 UTC 2015 ; Query time: 1836 msec Wed Jul 29 13:40:36 UTC 2015 ; Query time: 2182 msec Wed Jul 29 13:40:41 UTC 2015 ; Query time: 2079 msec Wed Jul 29 13:40:46 UTC 2015 ; Query time: 2066 msec Wed Jul 29 13:40:51 UTC 2015 ; Query time: 1905 msec Wed Jul 29 13:40:56 UTC 2015 ; Query time: 1999 msec Wed Jul 29 13:41:01 UTC 2015 ;

Query time: 1855 msec Wed Jul 29 13:41:06 UTC 2015 ; Query time: 2043 msec Wed Jul 29 13:41:11 UTC 2015 But it's not over, curl on ip gives me that : until [[ 0 -eq 1 ]]; do curl -I 10.0.126.115:9200 -s -w '%{time_total}' -o /dev/null | grep -E "^[1-9]" && date; done 083 Wed Jul 29 13:50:16 UTC 2015 247 Wed Jul 29 13:50:21 UTC 2015 874 Wed Jul 29 13:50:26 UTC 2015 019 Wed Jul 29 13:50:31 UTC 2015 006 Wed Jul 29 13:50:36 UTC 2015 718 Wed Jul 29 13:50:41 UTC 2015 954 Wed Jul 29 13:50:46 UTC 2015 065 Wed Jul 29 13:50:51 UTC 2015 896 Wed Jul 29 13:50:56 UTC 2015 968 Wed Jul 29 13:51:01 UTC 2015 556 Wed Jul 29 13:51:05 UTC 2015 There is something wrong in the way kube-proxy is working/updating. It's a real source of troubles on my side.
With further analysis anything run good on docker part, checking directly the virtual ip address of docker pod always return in time! checked on coreos and default kubernetes debian image.
Open question : Why using iptables to handle alone the inter-cluster communication instead of routing? each pod has it's own ip, it doesn't suffer from iptables refresh slowness, as well as missing rule. Nodes forwarding work great and on gce, it's ok to do so. Skydns might handle pod ip as RR dns A record and all that mess will run smoother. Currently I do have dbs with replication, php with some dependancies to other pods, so you can easily understand how high I'm dependant to a clean and really fast network.

I've got ~150 services wich lead to 2s with complete network stale, with 375 services it would take the whole refresh loop, and with more services what would happens? I've got the feeling that generating a complete ipfile and injecting with iprestore will do less harm. It's somehow important that kube team is looking at that. Production ready isn't really true on my case.
There is not enough information here to properly address your issue. What Kubernetes version, what OS, what cloud, how did you do setup, what network overlay if any, etc. Running on my own cluster, for example: and
you're not running enough samples, to be sure to have a least one pass on kube-proxy refresh. You'll have to run more than 5000 loops. That's why I filtered on time 1s. Good points on DNS RR. But it's the easiest way to handle multiple endpoints while real balancing is WIP. So I'm on a full 1.0.1 cluster with 1 master, 12 nodes : 11 coreos, 1 debian. The gce nodes are all on same power : n1-highmem-2. The installation is from a kube-up on release 1.0.1 package. 2 Modifications has been made on kubelet pkg to fit my need : a relocation of safeformatand_mount shell script and a safe parameter on iptables (-w switch) and that's all for coreos. The debian node is full stock, "only" there to perform debug tasks and analysis as I send here. The debian master is full stock too except the installation of ... htop . The tests provided is from the debian node, as I know it is the main OS. I've got: 104 pods (2 kubedns pods) 102 rc 101 svc 19 namespaces 17 external lb. No overlays , nothing special, I stood as close as possible to upstream, even on the coreos installation. We are not live now, so I still have time to debug anything you want and test whatever would make differences.
We know kube-proxy has a long-tail problem, but it's not usually multi-second. See the "histogram" below. Your point about doing iptables-restore in one shot is taken. root for i in $(seq 1 );

389 2 0.391 1 0.960 , bnprss wrote:
I was thinking more on iptables-restore, and I don't really see how to use it safely. There's no compare-and-swap optimistic concurrency mode and there's no way I can see to load just a single chain - only whole tables. This makes it very awkward to use for our "check if it exists and if not add it" mode, especially if anyone else is using the NAT table. Consider: kube-proxy reads state of -t nat kube-proxy iterates over state, checking for each rule it needs user modifies -t nat kube-proxy needs to make a change, replaces the -t nat section for just its own chain kube-proxy restores the state the user change is wiped out , Tim Hockin wrote:
I made some test, and I'm now sure: no traffic is handled at all while kube-proxy iptables loop, checked by logging on level 5 . I see three potential solutions : 1) keep the run whatever it cost Advantages: little changes Drawback: It will be a major problem with a lot of services, the check will take much more time than the syncloop allow. might be a huge CPU consummer 2) change the way kube-proxy check and insert rules step one : iptables-save step two : check rule existance step three : iptables inclusion of missing rule (like it does right now) Advantages: limit the rate of external program call, do the thing safely and keep custom rules by not using iptables-restore. I guess there will be some perf improvement by just parsing/checking one file. Drawback: none? 3) use external tool I've got in mind haproxy Advantages: All logic in the roadmap (and more) is allready implemented and well tested. Performance is really good. Drawback: dynamic inclustion/deletion of things must trig a reload of a map file... Network transaction shouldn't be stop anytime in the kube-proxy process.
I'll post the following PR that cover the 1st case : time to check : for i in $(seq 1 ); do curl -Ik 10.0.249.163 -s -w '%{time_total}' -o /dev/null; echo;

A better design would be store an atomic value in the ServiceInfo struct and simply check that value in each ProxyLoop(). I'm going to re-title this bug to reflect the problem , bnprss wrote:
It is really a simple ugly workaround focused on the big loop that takes ages to complete, I agree with that. And I assume it's mitigation role. My knowledge of go wont't help for your proposal, as you might have understood, it was the first time I code in that language and as a SysAdmin my skills are also limited in development tasks. So I can't handle such logic changes in an efficient way. Could you take care of it?
I'll see if I can throw something together. Do you want me to take over the lock mitigation too, or do you want to do that? It should be a relatively simple change given that you already found the place to do that.
No, it will be ok for the lock mitigation I figured out why red light cames from shippable and travis. And will comment on the PR. Thank you for taking care of the mid term solution.


Question: clock clock.Clock } <del> func (recorder *recorderImpl) generateEvent(object runtime.Object, annotations map[string]string, timestamp metav1.Time, eventtype, reason, message string) { </del> <ins> func (recorder *recorderImpl) generateEvent(object runtime.Object, annotations map[string]string, eventtype, reason, message string) { </ins> ref, err := ref.GetReference(recorder.scheme, object) if err != nil { klog.Errorf("Could not construct reference to: '%#v' due to: '%v'. Will not report event: '%v' '%v' '%v'", object, err, eventtype, reason, message)

is defined here but never used. As this is an unexported method it doesn't affect anyone, just a small point of confusion for the reader. /sig api-machinery /kind bug
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted </details
/triage accepted removing triage since the PR seem to be fixing the issue /assign Please close when merged! thank you


Question: f = framework.NewDefaultFramework("daemonsets") image := "gcr.io/google_containers/serve_hostname:v1.4" <del> redisImage := "gcr.io/google_containers/redis:e2e" </del> <ins> redisImage := "gcr.io/k8s-testimages/redis:e2e" </ins> dsName := "daemon-set" var ns string

Kubernetes version (use ): master What happened: Pulling the redis image takes a while, like 30 seconds in GCE. This might be leading to periodic timeouts in tests like . What you expected to happen: Tests shouldn't depend on large images that take a potentially long time to pull. proposed fix to reduce flakiness we can increase the replica count on this test to 2 or make the redis image we use smaller (or not use redis at all and just use a simpler smaller image)
cc (considering this as a bug since we shouldnt need a 500MG image just to test kubectl :))
This should be as simple as changing the current to use alpine and install redis, instead of basing it off debian, installing gcc, and compiling redis from source :-) The image should be ~10MB.


Question: config.DeleteNodePortService() ginkgo.By(fmt.Sprintf("dialing(udp) %v (node) --> %v:%v (nodeIP) and getting ZERO host endpoints", config.NodeIP, config.NodeIP, config.NodeUDPPort)) <del> err = config.DialFromNode("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, config.MaxTries, sets.NewString()) </del> <ins> // #106770 MaxTries can be very large on large clusters, with the risk that a new NodePort is created by another test and start to answer traffic. // Since we only want to assert that traffic is not being forwarded anymore and the retry timeout is 2 seconds, consider the test is correct // if the service doesn't answer after 10 tries. err = config.DialFromNode("udp", config.NodeIP, config.NodeUDPPort, 10, 10, sets.NewString()) </ins> if err != nil { framework.Failf("Failure validating that node port service STOPPED removed properly: %v", err) }

gce-master-scale-correctness Kubernetes e2e suite.[sig-network] Networking Granular Checks: Services should update nodePort: udp [Slow] ci-kubernetes-e2e-gce-scale-correctness.Overall kubetest.Test Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode should not mount / map unused volumes in a pod [LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Generic Ephemeral-volume (default fs) (late-binding)] ephemeral should support two pods which have the same volume definition Kubernetes e2e suite.[sig-network] Services should be able to update service type to NodePort listening on same port number but different protocols 12/1/2021 17:36:39 ET No response No response /sig storage /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.23 /priority important-soon
/assign aojea
/assign bowei
heh, what are the chances, two different test services took the same port The test was expecting to not receive any answer but the second service replies and makes the test fail link


Question: // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor <del> // Removes the given node from the record of attach updates. The node's entire // volumesToReportAsAttached list is removed. RemoveNodeFromAttachUpdates(nodeName types.NodeName) error </del> } // AttachedVolume represents a volume that is attached to a node.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: Deployment running with read-only GCE PD. A node containing one of the deployment pods is deleted and recreated soon after. New pod is scheduled to the same node, but is stuck in ContainerCreating state. The node's kubelet logs show continuous "Volume is not yet attached according to node" errors. What you expected to happen: Pod recreated successfully. Anything else we need to know?: Cause by PR The issue happens as follows: When the node is deleted, GCE PD is detached from the cloud provider. AttachDetachController discovers the PD has been detached, and marks it as detached. The original pod still exists in the API server (i.e. before garbage collection), so AttachDetachController reattaches the PD, successfully. Before the node recovers, ADC tries to update the node status but finds that the node doesn't exist. Following the logic introduced in it never attempts to update the node status again. The node recovers, a new pod is scheduled to it, waits for a successful attach signal in the node status but never gets it. /cc /sig storage
/assign


Question: }) }) <del> func generateServicesInNamespace(namespace string, num int) []*v1.Service { </del> <ins> func generateServicesInNamespaces(namespaces []string, num int) []*v1.Service { </ins> services := make([]*v1.Service, num) for i := 0; i < num; i++ { services[i] = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "svc-" + strconv.Itoa(i), <del> Namespace: namespace, </del> <ins> Namespace: namespaces[i%len(namespaces)], </ins> }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: For these 2 test suites visible in the testgrid at network-gce : http://k8s- http://k8s- the testsuite cannot setup correctly the cluster, due to a firewall issue (I think) See part of the build log below: What you expected to happen: I would think that setup the cluster is part of the process of the test-infra job, not the e2e test itself. Any missing flag or option on the job definition (see below) ? Need guidance on how to find the root cause of this error. How to reproduce it (as minimally and precisely as possible): Just look at the testgrid, it happen every 6 hours. http://k8s- http://k8s- Anything else we need to know?: job definition is: NOTE: this job definition was copied from another one that seems working fine: Environment: Kubernetes version (use ): current (branch master) Cloud provider or hardware configuration: gce
: I finally create this issue to get help. , : any idea on why the firewall rules cannot be set ? Could you indicate how to investigate this issue ? Thanks..
/sig network /sig scalability
I think it's because the name is too long. There's a limit of 64 chars or sth for resource names on GCP. Try changing the --cluster arg to or sth.
proposed a fix here : it is now merge. Need to verify the next run of these testsuites within 0 to 6 hours from now. : Thanks for your advise !
After updating the name of the cluster (shorter). The Cluster is now setup correctly. The PerformanceDNS test itself is failing - unable to create he 10k services. We believe that is an issue on the number of 10k services that k8s cannot support with this configuration of cluster. Next step is to decrease the number of services to 5k - a number we tested locally and know that k8s support.
Can you tell the reason why it's unable to create so many services?
I suspect that the cluster cannot operate correctly with that many services.

It can create the services. But the cluster acts undesirably after they are created. In my local tests on a single node cluster when I ramped up the number of services to 10K, I saw the same mode of failure. With 10K services, the cluster is unable to spin up the "util" pod. The "util" pod is required in the test to query the DNS from the context of a pod - if it's not running then the test cannot proceed.
But do you know what exactly is happening? What do you mean by "cluster is unable to spin up the "util" pod"? Is the pod not even being created? Or maybe is it scheduled, but kubelet is not starting it? Having more insight would help debugging it (maybe it's just size of master machine that is too small).
I dont for sure. But... According to the logs, I think the pod is successfully created. But when we try to run DNS queries, it's the container in the pod named "util" that cannot be found - not necessarily the pod itself. So, it seems the pod may be still starting up at the time - perhaps slowed by the number of services present. Maybe if I create the pod before creating the services it will help.
Let's try to understand this better. If it's that creating services is limiting us on the api qps, then starting the pod may potentially be slowed down (due to shortage of qps). Your idea sounds good, or you might alternatively try to slow down the rate of service creation. In general, I believe it shouldn't be a problem to create so many empty services in the cluster.
Please paste link to logs from that cluster (run from testgrid) - so that others can take a look into it.
OK, these are logs of the failure when pod is created after services... creates services OK I believe (this step takes just under 10 minutes). Then it creates the client pod... Looks OK? - client pod is created... So we start querying DNS... But the "util" container is not found, possibly because the pod is not ready yet. Perhaps the helper func that I am using doesn't wait for the pod to be ready.
I looked into kubelet logs - pods was succesfully started but then dies (crashed?) - something seems to be wrong on your side.

Logs from here:
It waits for it to be ready, but it dies after that.
OK, thanks but this isnt a pod I have invented for this test. It is already used for testing DNS in other existing tests. So I was expecting it to work. And it does work fine if i reduce the number of services - which I'll go ahead and do.
Let's not down the number of service without understanding why it fails. We need to understand that.
Does anyone know if the 10K limit on services is a hard/enforced limit? I didnt think it was (but thats just an assumption) - but if it is enforced somewhere, and exceeding it causes silent failures, then i'll need to reduce the number of services by a "few" to account for the other already existing services (e.g. the cluster dns service, api, etc...)
It's not a hard limit.
I managed to get this tested locally with 10K services on a single node.... where I create the pod before the services, and it passes. OK if I submit a PR with that change? Or do we want to first try to understand why creating the client pod after services causes a failure?
If you can explain why this works now and didn't work before, I'm fine - otherwise there is a risk that it may e.g. be flaky. We should understand it before blindly changing that.
OK - so I've run it and catched that pod for you. The reason why it is crashlooping is the following log: It just can't initialize itself due to too long argument list. - So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? - it's the limitation that recently poped up in our conversations ^^
Thanks Wojtek!

And that is why I said we should make use of the experiments dns folks are already doing, to catch that limitation (and we indeed seem to have caught it) :) On Thu, Jun 7, 2018, 10:47 AM Wojciech Tyczynski <:
I'll update that discussion once the value is figured out. Btw - thanks for digging. On Thu, Jun 7, 2018, 11:03 AM Shyam Jeedigunta <: The reason why it is crashlooping is the following log: exec user process caused "argument list too long" It just can't initialize itself due to too long argument list. So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? it's the limitation that recently poped up in our conversations ^^ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine.
But I got it to work with 10k services in one namespace... by creating the client pod before creating the services. On Jun 7, 2018, at 5:47 AM, Wojciech Tyczynski <mailto:: FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread
- IIUC that's because you created the pod even before creating the services (so it wasn't passed those service env vars while creation).
OK cool.

Seems like the limit would need to be here ... Several of the entries are "per ns" but there isnt one for services. As an unrelated side note: I just noticed that the limit on the number of deployments per ns in the thresholds doc lists 20K, with a long term goal of 10K. Is that a typo? Or is the long term goal to reduce the limit?
Sent for it. Seems so to me (should've probably been 10^5?). cc
Yeah - I think two values are incorrect wrt it: pods/namespace - I think we've never run 15k deployments/namespace - it doesn't make much sense to support more deployments than pods Will send out a PR for it.


Question: // CountIntolerableTaintsPreferNoSchedule gives the count of intolerable taints of a pod with effect PreferNoSchedule func countIntolerableTaintsPreferNoSchedule(taints []v1.Taint, tolerations []v1.Toleration) (intolerableTaints int) { <del> for i := range taints { taint := &taints[i] </del> <ins> for _, taint := range taints { </ins> // check only on taints that have effect PreferNoSchedule if taint.Effect != v1.TaintEffectPreferNoSchedule { continue } <del> if !v1.TolerationsTolerateTaint(tolerations, taint) { </del> <ins> if !v1.TolerationsTolerateTaint(tolerations, &taint) { </ins> intolerableTaints++ } } return } <del> // getAllTolerationEffectPreferNoSchedule gets the list of all Toleration with Effect PreferNoSchedule </del> <ins> // getAllTolerationEffectPreferNoSchedule gets the list of all Tolerations with Effect PreferNoSchedule or with no effect. </ins> func getAllTolerationPreferNoSchedule(tolerations []v1.Toleration) (tolerationList []v1.Toleration) { <del> for i := range tolerations { toleration := &tolerations[i] </del> <ins> for _, toleration := range tolerations { // Empty effect means all effects which includes PreferNoSchedule, so we need to collect it as well. </ins> if len(toleration.Effect) == 0 || toleration.Effect == v1.TaintEffectPreferNoSchedule { <del> tolerationList = append(tolerationList, *toleration) </del> <ins> tolerationList = append(tolerationList, toleration) </ins> } } return } <del> func getTolerationListFromPod(pod *v1.Pod) ([]v1.Toleration, error) { return getAllTolerationPreferNoSchedule(pod.Spec.Tolerations), nil } </del> // ComputeTaintTolerationPriority prepares the priority list for all the nodes based on the number of intolerable taints on the node func ComputeTaintTolerationPriorityMap(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (schedulerapi.HostPriority, error) { node := nodeInfo.Node() if node == nil { return schedulerapi.HostPriority{}, fmt.Errorf("node not found") } <del> var tolerationList []v1.Toleration </del> <ins> // To hold all the tolerations with Effect PreferNoSchedule var tolerationsPreferNoSchedule []v1.Toleration </ins> if priorityMeta, ok := meta.(*priorityMetadata); ok { <del> tolerationList = priorityMeta.podTolerations </del> <ins> tolerationsPreferNoSchedule = priorityMeta.podTolerations </ins> } else { <del> var err error tolerationList, err = getTolerationListFromPod(pod) if err != nil { return schedulerapi.HostPriority{}, err } </del> <ins> tolerationsPreferNoSchedule = getAllTolerationPreferNoSchedule(pod.Spec.Tolerations) </ins> } return schedulerapi.HostPriority{ Host:  node.Name, <del> Score: countIntolerableTaintsPreferNoSchedule(node.Spec.Taints, tolerationList), </del> <ins> Score: countIntolerableTaintsPreferNoSchedule(node.Spec.Taints, tolerationsPreferNoSchedule), </ins> }, nil }

Kubernetes version (use kubectl version): master What happened: Unnecessary always nil return value for function getTolerationListFromPod. What you expected to happen: We need to remove error return value for getTolerationListFromPod function.
/cc
cc


Question: // TODO: expose same flags as client.BindClientConfigFlags but for a server clientConfig := &client.Config{ <del> Host:    net.JoinHostPort(s.Address.String(), strconv.Itoa(int(s.Port))), </del> <ins> Host:    net.JoinHostPort(s.Address.String(), strconv.Itoa(s.WideOpenPort)), </ins> Version: s.StorageVersion, } client, err := client.New(clientConfig)

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: if port < 0 || port > math.MaxUint16 { return fmt.Errorf("invalid port %d", port) } <del> return portForward(r.client, podSandboxID, port, stream) </del> <ins> return r.portForward(podSandboxID, port, stream) </ins> } // ExecSync executes a command in the container, and returns the stdout output.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.


Question: } return nil <del> }, 2*time.Minute, time.Second*4).Should(gomega.BeNil()) </del> <ins> }, 30*time.Second, time.Second*4).Should(gomega.BeNil()) </ins> }) ginkgo.It("burstable container's oom-score-adj should be between [2, 1000)", func(ctx context.Context) {

Test: [sig-node] Restart [Serial] [Slow] [Disruptive] Kubelet should correctly account for terminated pods after restart I've also seen on /kind flake /sig node Looks like occasionally we have a second 50m CPU pod hanging around on the node, blocking enough pods getting scheduled for the test: I'm a bit puzzled because no pods are listed as currently running when the node dumps debug info. Could be a bug?
another case [sig-node] Container Manager Misc [Serial] Validate OOM score adjustments [NodeFeature:OOMScoreAdj] affected the Restart test case, see the kubelet log, the kubelet-container-manager guaranteede9d81584 pod was received almost simultaneously with the restart-test pod
Since at 0316 21:25:08, the container of pod guaranteede9d81584 has just been creating, but the test case [sig-node] Container Manager Misc [Serial] has already finished at 0316 21:25:06, so the test case check condition may be not strict
/triage accepted /assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details