Question: return fmt.Sprintf("%v:%v", backend.ServiceName, backend.ServicePort.String()) } <ins> func formatHosts(rules []extensions.IngressRule) string { list := []string{} max := 3 more := false for _, rule := range rules { if len(list) == max { more = true } if !more && len(rule.Host) != 0 { list = append(list, rule.Host) } } if len(list) == 0 { return "*" } ret := strings.Join(list, ",") if more { return fmt.Sprintf("%s + %d more...", ret, len(rules)-max) } return ret } func formatPorts(tls []extensions.IngressTLS) string { if len(tls) != 0 { return "80, 443" } return "80" } </ins> func printIngress(ingress *extensions.Ingress, w io.Writer, options PrintOptions) error { name := ingress.Name namespace := ingress.Namespace <del> hostRules := ingress.Spec.Rules </del> if options.WithNamespace { if _, err := fmt.Fprintf(w, "%st", namespace); err != nil { return err

We should show a nicely formatted map, with tls and ports. This 50% works. Today we have: Which works well enough, but printing a single Ingress involves hand managing tabs and newlines till the entire Ingress is printed, i.e: So when we start mixing different Ingress types, we get jarring output: This is because the columns are formatted per the top ingress and don't match the bottom one. The tabbed writer used by kubectl to align columns gets confused on the whole.
you can assign this to me, will fix this next-candidate.
thanks!
Sounds like we should keep it to a single line () and move everything else to describe. Makes this bug easier.
yeah, we will move this to a single line, this is the principle.
was this finished for 1.3?
the code is already finished on my local branch. I will open a PR as I confirmed it.
what do you think if print ingress like this: Also the original resource is:
That would fall right off the screen. I think we have to stick to just printing the ips in , and everything else in . You can print hoststs alone, and optimize for small ingresses by doing something like we do for endpoints (show a small number of characters then do "..."). The problem with trying to print everything on one line is you can't represent the fanout cleanly, ie you might do: But with multiple hosts: Doesn't work, because /bar and secret2 only applies to
I agree with this, maybe we should just remove these rules from get ingress output or just leave host here.
SGTM are you going to send a pr?
yes, ready to.


Question: # we will run provision to update code each time we test, so we do not want to do salt install each time if ! which salt-minion >/dev/null 2>&1; then # Install Salt <del> # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 curl -sS -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -X ## TODO this only works on systemd distros, need to find a work-around as removing -X above fails to start the services installed systemctl enable salt-minion systemctl start salt-minion </del> <ins> curl -sS -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s </ins> fi # run the networking setup

Jenkins build Node/job/heapster-dockercanarybuild-ci/58/ failed since last Thursday due to compiling issue: looks like it is caused by
encountered this error when compiling heapster at local.
After updating the gce util library Heapster requires now go in version = 1.5. I'll fix the version in Jenkins config. ref , ,
The problem is with: heapster-dockercanarybuild-ci heapster-gce-e2e-ci could you please bump go version to 1.6 for those Jenkins jobs?
how are these builds getting the Go runtime? Just whatever's installed on the system?
/usr/local/go
Configured here:
Ah, okay, it is whatever is installed on the system. We should probably change this to use something version-controlled. (Ideally this would just be running inside Docker.) I can upgrade to go 1.6 in the interim. I think the only other use of the system Go runtime is in the e2e tests, which shouldn't care about the version of go.
Updated to go1.6 on post-commit Jenkins VMs. Haven't touched PR Jenkins (yet).
in case you see any breakages, see ^
Thanks for the heads up
this is still failing. And actually looking at the latest build log, I see why: You need to update
Thanks I'll fix it.
Well, that's for , at least. is using system Go, and it's failing due to changes in Go 1.6, I think.
Log from :
seems to work. This used to happen before.
Suite heapster-gce-e2e-ci is still failing, this time with the message:
sorry, I missed that you pasted the same failure. I'll take w look.
has not passed in over a month. I'm going to send out a PR to disable it
As I wrote in the job has passed only once during its first run and then it failed 1000+ times. Was it supposed to work anytime?
If the solution is to delete this job that is totally fine by me
Decreasing priority since the job is disabled. do you know who set up the job?
do you know who created the suite? What should it test? The suite passed only 1/1175 runs.

had setup these jobs originally as part of migrating over to k8s jenkins server from my own jenkins server. The PR builder and HEAD CI should both be sharing the same framework ideally.
Thanks for the explanation. What is HEAD CI? I'll try to resurrect the suite. did you finish all the required work on this? Especially the suite passed only once and then failed 1000+ times.
I believe the suite was passing before the failures you see, but when it was converted to use jenkins-job-builder, the job was renamed and the count reset. It is possible the failures are related to something with the jjb change. After the initial migration from vish's jenkins server, I remember that they were passing for a while.
CI head is the "heapster-gce-e2e-ci" job running on the jenkins continuous integration server (as opposed to the jenkins PR builder server)
Thanks for clarification.
Let me know if it would be helpful to do a knowledge transfer about how the jenkins jobs are setup.
Removing flake label as this is not a flake.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig instrumentation because heapster /assign closing due to inactivity
/close


Question: qs.counter.Add(-1) } <ins> func (qs *queueSet) UpdateObservations() { qs.obsPair.RequestsWaiting.Add(0) qs.obsPair.RequestsExecuting.Add(0) } </ins> func (qs *queueSet) Dump(includeRequestDetails bool) debug.QueueSetDump { qs.lock.Lock() defer qs.lock.Unlock()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: } } <ins> func TestOwnerReferences(t *testing.T) { t.Parallel() trueVar := true falseVar := false refs := []metav1.OwnerReference{ { APIVersion: "v2", Kind:       "K2", Name:       "n2", UID:        types.UID("abc1"), }, { APIVersion:         "v1", Kind:               "K1", Name:               "n1", UID:                types.UID("abc2"), Controller:         &trueVar, BlockOwnerDeletion: &falseVar, }, { APIVersion:         "v3", Kind:               "K3", Name:               "n3", UID:                types.UID("abc3"), Controller:         &falseVar, BlockOwnerDeletion: &trueVar, }, } for i, ref := range refs { ref := ref t.Run(strconv.Itoa(i), func(t *testing.T) { t.Parallel() u1 := unstructured.Unstructured{ Object: make(map[string]interface{}), } refsX := []metav1.OwnerReference{ref} u1.SetOwnerReferences(refsX) have := u1.GetOwnerReferences() if !reflect.DeepEqual(have, refsX) { t.Errorf("Object references are not the same: %#v != %#v", have, refsX) } }) } } </ins> func TestUnstructuredListGetters(t *testing.T) { unstruct := unstructured.UnstructuredList{ Object: map[string]interface{}{

Kubernetes version Code from master. What happened: does not return and fields after they are set to non-nil values using . How to reproduce it (as minimally and precisely as possible): See PR
/sig api-machinery
This is a good find and I don't think I saw the notification come through :(


Question: # Install a default storage class (enabled by default) DEFAULT_STORAGE_CLASS=${KUBE_DEFAULT_STORAGE_CLASS:-true} <del> # start the cache mutation detector by default so that cache mutators will be found KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-true}" </del> <ins> # Do not run the mutation detector by default on a local cluster. # It is intended for a specific type of testing and inherently leaks memory. KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-false}" </ins> export KUBE_CACHE_MUTATION_DETECTOR # panic the server on watch decode errors since they are considered coder mistakes

/kind bug What happened: Start a cluster using ./hack/local- Now just monitor the memory on kube-apiserver and kube-controller-manager. Without doing anything else note the memory on each process has grown by 500 MB each day. Tried this for 5 days and saw this steady growth. What you expected to happen: Memory to remain relatively flat after some initial growth. How to reproduce it (as minimally and precisely as possible): Run ./hack/local- and wait. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10 Cloud provider or hardware configuration: local OS (e.g. from /etc/os-release): Debian GNU/Linux Kernel (e.g. ): Debian 4.9.65-3+deb9u2 Install tools: Others:
/sig api-machinery
/milestone v1.10
/priority important-soon
!
Copying some of what was discussed in Slack - we need to redo a test with set when running . This disables the cache mutation detector, which holds on to "all the things" .
What does the chart mean? yeah determined that most of it is from the mutation detector (it really makes me nervous to have test code linked in to the production binaries).
It was a differential pprof memory chart of hack/local-cluster- But as noted, we have to run it without mutation detection. I did that for an hour yesterday and did not notice much memory leaking.
After the suggested fix (turn of mutation detection), there was no noticeable increase on controller manager memory and the api-server memory increase was down to 8 MB per day. (on my setup)
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 2 days, the issue will be moved out of the v1.10 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
closing this to remove it from the release radar. Feel free to reopen if you think we still have an issue.


Question: } func getACRDockerEntryFromARMToken(a *acrProvider, loginServer string) (*credentialprovider.DockerConfigEntry, error) { <ins> // Run EnsureFresh to make sure the token is valid and does not expire if err := a.servicePrincipalToken.EnsureFresh(); err != nil { klog.Errorf("Failed to ensure fresh service principal token: %v", err) return nil, err } </ins> armAccessToken := a.servicePrincipalToken.OAuthToken() klog.V(4).Infof("discovering auth redirects for: %s", loginServer)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func createCgroupManager(name string) (*fs.Manager, error) { var memoryLimit uint64 memoryCapacity, err := getMemoryCapacity() <del> if err != nil || memoryCapacity*dockerMemoryLimitThresholdPercent/100 < minDockerMemoryLimit { </del> <ins> if err != nil { glog.Errorf("Failed to get the memory capacity on machine: %v", err) } else { memoryLimit = memoryCapacity * dockerMemoryLimitThresholdPercent / 100 } if err != nil || memoryLimit < minDockerMemoryLimit { </ins> memoryLimit = minDockerMemoryLimit } glog.V(2).Infof("Configure resource-only container %q with memory limit: %d", name, memoryLimit)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The log from kubelet service: 07:41:58. 4036 ] Configure resource-only container "" with memory limit: 0 Environment: Kubernetes version (use ):v1.10.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig kubelet cc
/sig node /area kubelet


Question: import ( "fmt" <ins> "time" </ins> dockerref "github.com/docker/distribution/reference" v1 "k8s.io/api/core/v1"

What would you like to be : The 'Successfully pulled image' event should contain the duration for image pulling. Why is this needed: Currently the success event is in this form: User has to trace back to the corresponding event for the beginning of image pulling and calculate the duration him/herself:
/sig node /assign


Question: var ( applyConfiguration = types.Ref("k8s.io/apimachinery/pkg/runtime", "ApplyConfiguration") groupVersionKind   = types.Ref("k8s.io/apimachinery/pkg/runtime/schema", "GroupVersionKind") <ins> typeMeta           = types.Ref("k8s.io/apimachinery/pkg/apis/meta/v1", "TypeMeta") </ins> objectMeta         = types.Ref("k8s.io/apimachinery/pkg/apis/meta/v1", "ObjectMeta") rawExtension       = types.Ref("k8s.io/apimachinery/pkg/runtime", "RawExtension") unknown            = types.Ref("k8s.io/apimachinery/pkg/runtime", "Unknown")

Most of the FooApplyConfiguration constructors pre-fill the Kind and APIVersion fields to make using them smooth. But () is missing this, making patterns like this fail: Currently this code fails with (server-side) error . The constructor should set the minimum required fields for the resulting object to be usable with calls, similar to how e.g. the `D Create a deployment (however) Create a clientset, and get the Deployments client from it Try to call deployments.ApplyScale using a ScaleApplyConfiguration created by the Scale() constructor I can see that the presence or absence of these metadata fields is driven by some code generation, but I wasn't able to follow through to what configures that code generation. I assume it's something in the schemas somewhere, but that wasn't clear to me. <details</details<detailsminikube v1.25.1 </detailsN/A N/A N/A N/A
/sig api-machinery
/assign /triage accepted
/assign


Question: return "", err } <del> // Search for attachment by VolumeAttachment.Spec.Source.PersistentVolumeName nodeName := string(m.plugin.host.GetNodeName()) attachID := getAttachmentName(csiSource.VolumeHandle, csiSource.Driver, nodeName) attachment, err := m.k8s.StorageV1().VolumeAttachments().Get(attachID, meta.GetOptions{}) </del> <ins> driverName := csiSource.Driver skip, err := m.plugin.skipAttach(driverName) </ins> if err != nil { <del> klog.Error(log("blockMapper.SetupDevice failed to get volume attachment [id=%v]: %v", attachID, err)) </del> <ins> klog.Error(log("blockMapper.SetupDevice failed to check CSIDriver for %s: %v", driverName, err)) </ins> return "", err } <del> if attachment == nil { klog.Error(log("blockMapper.SetupDevice unable to find VolumeAttachment [id=%s]", attachID)) return "", errors.New("no existing VolumeAttachment found") </del> <ins> var attachment *storage.VolumeAttachment if !skip { // Search for attachment by VolumeAttachment.Spec.Source.PersistentVolumeName nodeName := string(m.plugin.host.GetNodeName()) attachID := getAttachmentName(csiSource.VolumeHandle, csiSource.Driver, nodeName) attachment, err = m.k8s.StorageV1().VolumeAttachments().Get(attachID, meta.GetOptions{}) if err != nil { klog.Error(log("blockMapper.SetupDevice failed to get volume attachment [id=%v]: %v", attachID, err)) return "", err } </ins> } //TODO (vladimirvivien) implement better AccessModes mapping between k8s and CSI

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: During I develop a driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. PS: I use cluster-driver register and make requiredAttach as false. kubectl get csidriver csi-xxx -o yaml ! What you expected to happen: Mount a raw block device successfully. How to reproduce it (as minimally and precisely as possible): driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. cluster-driver register and make requiredAttach as false. a pod with raw block devices Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: BaiduCloud OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /assign
I would send a PR later


Question: return false, nil } <ins> // podSucceeded returns true if the pod has run to completion, false if the pod has not yet // reached running state, or an error in any other case. func podSucceeded(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: return false, errors.NewNotFound(schema.GroupResource{Resource: "pods"}, "") } switch t := event.Object.(type) { case *corev1.Pod: return t.Status.Phase == corev1.PodSucceeded, nil } return false, nil } </ins> // podRunningAndReady returns true if the pod is running and ready, false if the pod has not // yet reached those states, returns ErrPodCompleted if the pod has run to completion, or // an error in any other case.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: } var instances []string for _, instance := range res.Items { <del> instances = append(instances, instance.Name+suffix) </del> <ins> instances = append(instances, instance.Name) </ins> } return instances, nil }

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: Usage: ```console <del> kubectl exec test-agnhost -- /agnhost netexec [--http-port <http-port>] [--udp-port <udp-port>] [--sctp-port <sctp-port>] </del> <ins> kubectl exec test-agnhost -- /agnhost netexec [--http-port <http-port>] [--udp-port <udp-port>] [--sctp-port <sctp-port>] [--tls-cert-file <cert-file>] [--tls-private-key-file <privkey-file>] </ins> ``` ### nettest

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: unmountedVolumes = append(unmountedVolumes, expectedVolume) } } <ins> sort.Strings(unmountedVolumes) </ins> return unmountedVolumes }

Currently I describe a Pod that is in and fails to attach/mounts its volume: Note that the generated events are shown in 4 section. The only difference between the messages is how the are ordered. I suggest to sort the unmounted/unattached) volumes for the event message. This will allow to group all events and properly have the age and the occurrences count reported. I would expect the unmounted/unattached) volumes to be sorted in the events' message and to do not have several group of events for the same issue. See above. No response <details</details<detailsAWS but does not matter for this issue. </details<details</details<details</details<details</details<details</details
/sig storage
/assign qingwave
/triage accepted


Question: } // Launch a Replication Controller and wait for all pods it spawns <del> // to become running </del> <ins> // to become running.  The controller will need to be cleaned up external // to this method </ins> func RunRC(c *client.Client, name string, ns, image string, replicas int) { defer GinkgoRecover()

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: // When the last pending initializer is removed, and no failing result is set, the initializers // struct will be set to nil and the object is considered as initialized and visible to all // clients. <del> Pending []Initializer `json:"pending" protobuf:"bytes,1,rep,name=pending"` </del> <ins> // +patchMergeKey=name // +patchStrategy=merge Pending []Initializer `json:"pending" protobuf:"bytes,1,rep,name=pending" patchStrategy:"merge" patchMergeKey:"name"` </ins> // If result is set with the Failure field, the object will be persisted to storage and then deleted, // ensuring that other clients can observe the deletion. Result *Status `json:"result,omitempty" protobuf:"bytes,2,opt,name=result"`

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: } return nil } else if claim.Spec.VolumeName == "" { <ins> if isMisMatch, err := checkVolumeModeMisMatches(&claim.Spec, &volume.Spec); err != nil || isMisMatch { // Binding for the volume won't be called in syncUnboundClaim, // because findBestMatchForClaim won't return the volume due to volumeMode mismatch. volumeMsg := fmt.Sprintf("Cannot bind PersistentVolume to requested PersistentVolumeClaim %q due to incompatible volumeMode.", claim.Name) ctrl.eventRecorder.Event(volume, v1.EventTypeWarning, events.VolumeMismatch, volumeMsg) claimMsg := fmt.Sprintf("Cannot bind PersistentVolume %q to requested PersistentVolumeClaim due to incompatible volumeMode.", volume.Name) ctrl.eventRecorder.Event(claim, v1.EventTypeWarning, events.VolumeMismatch, claimMsg) // Skipping syncClaim return nil } </ins> if metav1.HasAnnotation(volume.ObjectMeta, annBoundByController) { // The binding is not completed; let PVC sync handle it glog.V(4).Infof("synchronizing PersistentVolume[%s]: volume not bound yet, waiting for syncClaim to fix it", volume.Name)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a user specifies volumeMode: Block in creating PVC and neither plug-in nor external provisioner doesn't support block, a PV is created, but is not bound to the PVC because the PV and PVC don't match. So, PVC will stay pending forever without any error messages. What you expected to happen: In above case, a proper error message is shown to users to make them understand what's wrong. How to reproduce it (as minimally and precisely as possible): up plug-in and external provisioner that don't support block. PVC using volumeMode: Block. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: KVM OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: hack/local-up- Others: /sig storage


Question: driver := NewDriver(f, nodes, func() app.Resources { return app.Resources{ NodeLocal:      true, <del> MaxAllocations: 2, </del> <ins> MaxAllocations: 1, </ins> Nodes:          nodes.NodeNames, } }) driver2 := NewDriver(f, nodes, func() app.Resources { return app.Resources{ NodeLocal:      true, <del> MaxAllocations: 2, </del> <ins> MaxAllocations: 1, </ins> Nodes:          nodes.NodeNames, AllocateWrapper: func(

What happened: Shortly after was merged, the "multiple nodes reallocation works" E2E DRA test started to fail randomly. Please provide links to example occurrences, if any: /sig node /kind bug needs to stay green. No response master n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, health: probe.Failure, }, <ins> { handler: redirectHandler(http.StatusMovedPermanently, false), // 301 health:  probe.Success, }, { handler: redirectHandler(http.StatusMovedPermanently, true), // 301 health:  probe.Failure, }, { handler: redirectHandler(http.StatusFound, false), // 302 health:  probe.Success, }, { handler: redirectHandler(http.StatusFound, true), // 302 health:  probe.Failure, }, { handler: redirectHandler(http.StatusTemporaryRedirect, false), // 307 health:  probe.Success, }, { handler: redirectHandler(http.StatusTemporaryRedirect, true), // 307 health:  probe.Failure, }, { handler: redirectHandler(http.StatusPermanentRedirect, false), // 308 health:  probe.Success, }, { handler: redirectHandler(http.StatusPermanentRedirect, true), // 308 health:  probe.Failure, }, </ins> } for i, test := range testCases { func() {

It seems the http liveness probe will not follow a 302 redirect and instead reports a failure. I found this in some testing on 1.0.6 and have not yet verified the same behavior on latest. Do you agree that it should check the redirected path on a 302? If so, where would be the best place to check the response code and handle this?
Why would you set your health-check URL to a redirect? , Dan Wilson wrote:
In this case it was a versioned health check where /v1/health or /v2/health is the correct check and /health directs to the correct one.
hmm, I could buy following redirects, I guess. It's a little weird if it redirects off-host, but who am I to say it is wrong? , Dan Wilson wrote:
Following redirects for a health check seems weird to me as well... I definitely wouldn't expect it to by default.
Seems like it would be cleaner to make respond the same as instead of making all health consumers follow redirects.
I agree that it's cleaner to call /health and make that route the same as /<latest/health but due to the framework used for this project it's not handled that way. Following the HTTP standards is important as well, right? curl handles this with an additional option (-L), could we do the same by adding a parameter?
I think it should follow local redirects without complaining (can well differentiate local vs non?). The real question is non-local redirects - should it a) work, b) not work, c) work iff a flag-field in the probe , Dan Wilson wrote:
old thread here... I agree with non-local redirects not being allowed, just doesn't seem to make sense. If we get a 301 perhaps we could get the value of the redirect ( it looks like there is a Location header with the new full path ) and then do something like the following: the root URL of the redirect location. it's a DNS name attempt to resolve it to an IP. If that fails, just bail. the IP to make sure it's in the cluster. If it's not in our CIDR block bail. the redirect. thoughts?

/sig node /sig network
Adding to 1.8 milestone
Is it not possible and easier to simply empower the user by adding an option for "unexceptional" status codes? They could declare 200-only (default) or all-but-errors.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: priority: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This contradicts docs , opened


Question: }, }, }, <ins> podIPs:          []string{"1.2.3.4", "fd00::6"}, </ins> masterServiceNs: "nothing", nilLister:       true, expectedEnvs: []kubecontainer.EnvVar{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: } } <ins> func TestGetIPFromLoopbackInterface(t *testing.T) { testCases := []struct { tcase      string family     AddressFamily nw         networkInterfacer expected   net.IP errStrFrag string }{ {"ipv4", familyIPv4, linkLocalLoopbackNetworkInterface{}, net.ParseIP("10.1.1.1"), ""}, {"ipv6", familyIPv6, linkLocalLoopbackNetworkInterface{}, net.ParseIP("fd00:1:1::1"), ""}, {"no global ipv4", familyIPv4, loopbackNetworkInterface{}, nil, ""}, {"no global ipv6", familyIPv6, loopbackNetworkInterface{}, nil, ""}, } for _, tc := range testCases { ip, err := getIPFromLoopbackInterface(tc.family, tc.nw) if err != nil { if !strings.Contains(err.Error(), tc.errStrFrag) { t.Errorf("case[%s]: Error string %q does not contain %q", tc.tcase, err, tc.errStrFrag) } } else if tc.errStrFrag != "" { t.Errorf("case[%s]: Error %q expected, but seen %v", tc.tcase, tc.errStrFrag, err) } else if !ip.Equal(tc.expected) { t.Errorf("case[%v]: expected %v, got %+v .err : %v", tc.tcase, tc.expected, ip, err) } } } </ins> func TestChooseHostInterfaceFromRoute(t *testing.T) { testCases := []struct { tcase    string

What happened: When attempting to kubeadm join a control plane node or worker on a machine with global IPv4 unicast addresses on a loopback address (router to the host w/ BGP Unnumbered), the join fails with the following output: See this issue for details : Also: What you expected to happen: The correct IPv4 address should be selected from the loopback adapter, or we should be able to specify an IPv4 address to use for egress. The IPv6 addresses on the network adapters themselves are only used to establish upstream peering. IPv4 nexthops are fake 169.254.x.x addresses artificially programmed via RFC5549 into the ARP table. How to reproduce it (as minimally and precisely as possible): On a node with the following conditions: Routing to the host with FRR IPv6 link locals on physical network interfaces No IPv4 addresses on physical network interfaces Unicast IPv4 on the loopback adapter IPv4 default route present via RFC5549 Attempt to kubeadm join a node, and it will fail. Anything else we need to know?: We've been patching kubeadm with , which works as expected. There was a fix applied to allow initing control plane nodes w/ bind address 127.0.0.1 which would then select an IPv4 from the loopback, but it doesn't appear to have fixed the joining issues. Pinging and by request on kubeadm issues. Environment: Kubernetes version (use ): 1.17.12 Cloud provider or hardware configuration: baremetal OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1127.19.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): Calico Others: FRR for peering with routers on physical network interfaces
/sig api-machinery
is defined but we should note the same function is used for the --bind-address of the kube-apiserver. /sig network
/assign
/triage accepted


Question: sort.Sort(sort.Reverse(sort.StringSlice(dk.index))) } <ins> const defaultRegistryHost = "index.docker.io/v1/" </ins> func (dk *dockerKeyring) lookup(image string) (docker.AuthConfiguration, bool) { // range over the index as iterating over a map does not provide // a predictable ordering

Pulls of images from private repos hosted on the default registry, , fail because the images are not prefixed, thus the fails. I've figured out a hacky way of getting around this issue by manually editing the .dockercfg but as a more permanent solution, I'd like to propose using authentication with the default registry if it is provided in the .dockercfg. Opening this issue to start some discussion. I'll attach a first pass at this in a pull request.


Question: - tallclair - MrHohn labels: <del> - sig/gcp </del> <ins> - area/provider/gcp </ins>

Some owners files attempt to auto apply the no longer existing label, which is now : This is present in 4 owners files: /assign /area oncall-hotlist
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package gce import ( "net" "reflect" "testing" compute "google.golang.org/api/compute/v1" ) func TestLastIPInRange(t *testing.T) { for _, tc := range []struct { cidr string want string }{ {"10.1.2.3/32", "10.1.2.3"}, {"10.1.2.0/31", "10.1.2.1"}, {"10.1.0.0/30", "10.1.0.3"}, {"10.0.0.0/29", "10.0.0.7"}, {"::0/128", "::"}, {"::0/127", "::1"}, {"::0/126", "::3"}, {"::0/120", "::ff"}, } { _, c, err := net.ParseCIDR(tc.cidr) if err != nil { t.Errorf("net.ParseCIDR(%v) = _, %v, %v; want nil", tc.cidr, c, err) continue } if lastIP := lastIPInRange(c); lastIP.String() != tc.want { t.Errorf("LastIPInRange(%v) = %v; want %v", tc.cidr, lastIP, tc.want) } } } func TestSubnetsInCIDR(t *testing.T) { subnets := []*compute.Subnetwork{ { Name:        "A", IpCidrRange: "10.0.0.0/20", }, { Name:        "B", IpCidrRange: "10.0.16.0/20", }, { Name:        "C", IpCidrRange: "10.132.0.0/20", }, { Name:        "D", IpCidrRange: "10.0.32.0/20", }, { Name:        "E", IpCidrRange: "10.134.0.0/20", }, } expectedNames := []string{"C", "E"} gotSubs, err := subnetsInCIDR(subnets, autoSubnetIPRange) if err != nil { t.Errorf("autoSubnetInList() = _, %v", err) } var gotNames []string for _, v := range gotSubs { gotNames = append(gotNames, v.Name) } if !reflect.DeepEqual(gotNames, expectedNames) { t.Errorf("autoSubnetInList() = %v, expected: %v", gotNames, expectedNames) } } </ins>

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: // TODO(jnagal): Infer schema through reflection. (See bigquery/client/example) func (self *bigqueryStorage) GetSchema() *bigquery.TableSchema { <del> fields := make([]*bigquery.TableFieldSchema, 18) </del> <ins> fields := make([]*bigquery.TableFieldSchema, 19) </ins> i := 0 fields[i] = &bigquery.TableFieldSchema{ Type: typeTimestamp,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Following link on sig-network page () is broken Link should be working Kubernetes version (use ):1.21 Cloud provider or hardware configuration:None OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig-network
/sig network
/triage accepted


Question: } // Check that ID list is correct. <del> ids := store.Contains() </del> <ins> ids := store.ContainedIDs() </ins> if !ids.HasAll("foo", "bar") { t.Errorf("missing items") }

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: ### Create redis-master service from JSON # Pre-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> # Command kubectl create -f examples/guestbook/redis-master-service.json "${kube_flags[@]}" # Post-condition: redis-master service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:' </ins> ### Dump current redis-master service output_service=$(kubectl get service redis-master -o json --output-version=v1beta3 "${kube_flags[@]}") ### Delete redis-master-service by id # Pre-condition: redis-master service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:' </ins> # Command kubectl delete service redis-master "${kube_flags[@]}" # Post-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> ### Create redis-master-service from dumped JSON # Pre-condition: Only the default kubernetes services are running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:' </ins> # Command echo "${output_service}" | kubectl create -f - "${kube_flags[@]}" # Post-condition: redis-master service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:' </ins> ### Create redis-master-${version}-test service # Pre-condition: redis-master-service service is running <del> kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:' </del> <ins> kube::test::get_object_assert services "{{range.items}}{{.$id_field}}:{{end}}" 'kubernetes:kubernetes-ro:redis-master:' </ins> # Command kubectl create -f - "${kube_flags[@]}" << __EOF__ {

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: network_mode: openvswitch node_ip: $MINION_IP etcd_servers: $MASTER_IP <ins> networkInterfaceName: eth1 apiservers: $MASTER_IP </ins> roles: - kubernetes-pool - kubernetes-pool-vagrant

The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver.

I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.


Question: It also assumes that ```$DOCKER_USER``` is set to your docker user id. <ins> You may need to open the firewall for port 8080 using the [console][cloud-console] or the `gcutil` tool. The following command will allow traffic from any source to instances tagged `kubernetes-minion`: ```shell $ gcutil addfirewall --allowed=tcp:8080 --target_tags=kubernetes-minion kubernetes-minion-8080 ``` </ins> ### Step One: Build the image $ cd kubernetes/examples/update-demo/image

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: $config = $FLUENTD_CONFIG.replace('NODE_NAME', (hostname)) $config | Out-File -FilePath $fluentd_config_file -Encoding ASCII Log-Output "Wrote fluentd logging config to $fluentd_config_file" <ins> # Configure StackdriverLogging to automatically restart on failure after 10 # seconds. The logging agent may die die to various disruptions but can be # resumed. sc.exe failure StackdriverLogging reset= 0 actions= restart/1000/restart/10000 </ins> } # The NODE_NAME placeholder must be replaced with the node's name (hostname).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There's a race condition between starting Windows service and when the VMs network is actually available to communicate with the VM's metadata server. If the service starts first and cannot communicate with the metadata server or start talking to the stackdriver logging service it'll crash and not resume because the service is not configured to restart on failure. Typically a is used for a service like this but Windows does not support privileged containers so a standard Windows Service is used. What you expected to happen: If the service fails to start it should be automatically be restarted. How to reproduce it (as minimally and precisely as possible): This is a flaky issue but a way to repro this is to effectively by RDP'ing into the node and running: Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 and onward Cloud provider or hardware configuration: GCE OS (e.g: ): Windows Kernel (e.g. ): Windows (any version) Install tools: cluster/gce/windows/README-GCE-Windows-kube- Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign jeremyedwards
GitHub didn't allow me to assign the following users: jeremyedwards. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign jeremyedwards Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } type vaTest struct { <del> testName          string volName           string podName           string podNodeName       string pvName            string vaName            string vaNodeName        string vaAttachStatus    bool csiMigration      bool expected_attaches map[string][]string expected_detaches map[string][]string </del> <ins> testName               string volName                string podName                string podNodeName            string pvName                 string vaName                 string vaNodeName             string vaAttachStatus         bool csiMigration           bool expected_attaches      map[string][]string expected_detaches      map[string][]string expectedASWAttachState cache.AttachState </ins> } func Test_ADC_VolumeAttachmentRecovery(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g.

because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted


Question: w.initialValue = results.Success } <del> w.proberResultsMetricLabels = prometheus.Labels{ "probe_type":     w.probeType.String(), "container_name": w.container.Name, "container":      w.container.Name, "pod_name":       w.pod.Name, "pod":            w.pod.Name, "namespace":      w.pod.Namespace, "pod_uid":        string(w.pod.UID), </del> <ins> basicMetricLabels := prometheus.Labels{ "probe_type": w.probeType.String(), "container":  w.container.Name, "pod":        w.pod.Name, "namespace":  w.pod.Namespace, "pod_uid":    string(w.pod.UID), </ins> } <ins> w.proberResultsSuccessfulMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsSuccessfulMetricLabels["result"] = probeResultSuccessful w.proberResultsFailedMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsFailedMetricLabels["result"] = probeResultFailed w.proberResultsUnknownMetricLabels = deepCopyPrometheusLabels(basicMetricLabels) w.proberResultsUnknownMetricLabels["result"] = probeResultUnknown </ins> return w }

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package editor import ( "errors" "fmt" "os" "os/exec" "path/filepath" "strings" "syscall" "k8s.io/klog/v2" "k8s.io/kubectl/pkg/util/term" ) // Enclose argument in double-quotes. Double each double-quote character as // an escape sequence. func cmdQuoteArg(arg string) string { var result strings.Builder result.WriteString(`"`) result.WriteString(strings.ReplaceAll(arg, `"`, `""`)) result.WriteString(`"`) return result.String() } func (e Editor) args(path string) []string { args := make([]string, len(e.Args)) copy(args, e.Args) if e.Shell { last := args[len(args)-1] if args[0] == windowsShell { // Use double-quotation around whole command line string // See https://stackoverflow.com/a/6378038 args[len(args)-1] = fmt.Sprintf(`"%s %s"`, last, cmdQuoteArg(path)) } else { args[len(args)-1] = fmt.Sprintf("%s %q", last, path) } } else { args = append(args, path) } return args } // Launch opens the described or returns an error. The TTY will be protected, and // SIGQUIT, SIGTERM, and SIGINT will all be trapped. func (e Editor) Launch(path string) error { if len(e.Args) == 0 { return fmt.Errorf("no editor defined, can't open %s", path) } abs, err := filepath.Abs(path) if err != nil { return err } args := e.args(abs) var cmd *exec.Cmd if e.Shell && args[0] == windowsShell { // Pass all arguments to cmd.exe as one string // See https://pkg.go.dev/os/exec#Command cmd = exec.Command(args[0]) cmd.SysProcAttr = &syscall.SysProcAttr{} cmd.SysProcAttr.CmdLine = strings.Join(args, " ") } else { cmd = exec.Command(args[0], args[1:]...) } cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr cmd.Stdin = os.Stdin klog.V(5).Infof("Opening file with editor %v", args) if err := (term.TTY{In: os.Stdin, TryDev: true}).Safe(cmd.Run); err != nil { if errors.Is(err, exec.ErrNotFound) { return fmt.Errorf("unable to launch the editor %q", strings.Join(e.Args, " ")) } return fmt.Errorf("there was a problem with the editor %q", strings.Join(e.Args, " ")) } return nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--I was trying to use a different editor than notepad on windows to be used by by setting the environment variable. Doing this was non trivial and the following did not work all resulting in different errors. What happened: What you expected to happen: I should be able to use normal windows paths, either by quoting them, or not. the workaround is to use a short filename however short paths are optional and may not always be created. () How to reproduce it (as minimally and precisely as possible): try and set an editor on windows where there is a space in the name of the command. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: N/A this is client side issue OS (e.g. from /etc/os-release): windows 10 1803 Kernel (e.g. ): :-) Install tools: Others:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
In case anyone wants an answer, it needs two things: (or ) into system PATH environment variable EDITOR=notepad2 -w (or EDITOR=code -w)
Currently there is no way how to pass path with spaces on Windows with or environment variables. Pull Request addresses this issue. With the fix, you would be able to specify the following: and when is in your , you could do as well: or (this is current workaround for the problem)


Question: networkingapiv1 "k8s.io/api/networking/v1" policyapiv1beta1 "k8s.io/api/policy/v1beta1" rbacv1 "k8s.io/api/rbac/v1" <del> rbacv1alpha1 "k8s.io/api/rbac/v1alpha1" </del> rbacv1beta1 "k8s.io/api/rbac/v1beta1" <del> schedulingapiv1alpha1 "k8s.io/api/scheduling/v1alpha1" settingv1alpha1 "k8s.io/api/settings/v1alpha1" </del> storageapiv1 "k8s.io/api/storage/v1" storageapiv1beta1 "k8s.io/api/storage/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

Need to follow-up on and i assigned it to you because turning off settings/v1alpha1 requires some changes in the tests.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/assign
did you mean I should add a priority label?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: # Set default values with override if [[ "${CONTAINER_RUNTIME}" == "docker" ]]; then <del> export CONTAINER_RUNTIME_ENDPOINT=${KUBE_CONTAINER_RUNTIME_ENDPOINT:-unix:///var/run/docker.sock} </del> <ins> export CONTAINER_RUNTIME_ENDPOINT=${KUBE_CONTAINER_RUNTIME_ENDPOINT:-unix:///var/run/dockershim.sock} </ins> export CONTAINER_RUNTIME_NAME=${KUBE_CONTAINER_RUNTIME_NAME:-docker} export LOAD_IMAGE_COMMAND=${KUBE_LOAD_IMAGE_COMMAND:-} elif [[ "${CONTAINER_RUNTIME}" == "containerd" ]]; then

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig testing /cc /priority critical-urgent /milestone v1.19
/assign
I see the PR merged, however, the job still failing in the recent run 22:31 PDT Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:44.597] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.070] scp: No such file or directory W0611 06:08:46.070] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.073] scp: No such file or directory W0611 06:08:46.074] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.079] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.084] ERROR: () [/usr/bin/scp] exited with return code [1].

W0611 06:08:51.917] INSTANCEGROUPS=bootstrap-e2e-minion-group W0611 06:08:51.918] NODENAMES=bootstrap-e2e-minion-group-dq3m bootstrap-e2e-minion-group-gjk0 bootstrap-e2e-minion-group-qpxw I0611 06:08:54.017] Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:56.191] 2020/06/11 06:08:56 Running: ./hack/e2e-internal/e2e-
I see the issue closed by the PR however the job still failing since 06-10 16:31 PDT
looks like we just turned green :)
cool! :)
Yes it's indeed ... thanks for the support!


Question: }) // createBalancedPodForNodes creates a pod per node that asks for enough resources to make all nodes have the same mem/cpu usage ratio. <del> func createBalancedPodForNodes(f *framework.Framework, cs clientset.Interface, ns string, nodes []v1.Node, requestedResource *v1.ResourceRequirements, ratio float64) { </del> <ins> func createBalancedPodForNodes(f *framework.Framework, cs clientset.Interface, ns string, nodes []v1.Node, requestedResource *v1.ResourceRequirements, ratio float64) error { </ins> // find the max, if the node has the max,use the one, if not,use the ratio parameter <del> var maxCpuFraction, maxMemFraction float64 = ratio, ratio </del> <ins> var maxCPUFraction, maxMemFraction float64 = ratio, ratio </ins> var cpuFractionMap = make(map[string]float64) var memFractionMap = make(map[string]float64) for _, node := range nodes { cpuFraction, memFraction := computeCpuMemFraction(cs, node, requestedResource) cpuFractionMap[node.Name] = cpuFraction memFractionMap[node.Name] = memFraction <del> if cpuFraction > maxCpuFraction { maxCpuFraction = cpuFraction </del> <ins> if cpuFraction > maxCPUFraction { maxCPUFraction = cpuFraction </ins> } if memFraction > maxMemFraction { maxMemFraction = memFraction } } // we need the max one to keep the same cpu/mem use rate <del> ratio = math.Max(maxCpuFraction, maxMemFraction) </del> <ins> ratio = math.Max(maxCPUFraction, maxMemFraction) </ins> for _, node := range nodes { memAllocatable, found := node.Status.Allocatable["memory"] Expect(found).To(Equal(true))

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: MaxContainerCount:       *maxContainerCount, Runonce:                 *runonce, Port:                    *port, <ins> CAdvisorPort:            *cAdvisorPort, </ins> EnableServer:            *enableServer, EnableDebuggingHandlers: *enableDebuggingHandlers, DockerClient:            kubelet.ConnectToDockerOrDie(*dockerEndpoint),

Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?


Question: enableNonPreempting bool <del> profiles         []schedulerapi.KubeSchedulerProfile registry         framework.Registry nodeInfoSnapshot *internalcache.Snapshot extenders        []schedulerapi.Extender </del> <ins> profiles          []schedulerapi.KubeSchedulerProfile registry          framework.Registry nodeInfoSnapshot  *internalcache.Snapshot extenders         []schedulerapi.Extender frameworkCapturer FrameworkCapturer </ins> } func (c *Configurator) buildFramework(p schedulerapi.KubeSchedulerProfile) (framework.Framework, error) { <ins> if c.frameworkCapturer != nil { c.frameworkCapturer(p) } </ins> return framework.NewFramework( c.registry, p.Plugins,

has compatibility tests for Policy v1. We are lacking similar tests for other options of , such as: [x] : This can be right away to test that default plugins can be disabled, changed order, etc. [x] : This one is trickier, as it requires a new method in to expose all configurations and a new method in so that they can say which args they used. /sig scheduling /help /kind cleanup
/assign
you can work on the first one. I'll have a PR soon that should clarify how the second one should be done.
I have sent a pr for the first one. And I'm working on the second one, I think it's better to keep 2 PRs for this issue? If so, PTAL at the first one, thanks.
Yes, 2 PRs is ideal. I'll review later today.
I have one small question about do I need to mock some fake plugins for testing , or just testing the default plugins?
You should use the existing plugins, because these are compatibility tests.
Okay
Actually, I think adding a new method such as to interface is a little redundant. In fact, most plugins don't need any argument for building them, so we are talking about adding a new method that needs to be implemented by all of the existing and coming plugins. And users who write out-of-tree plugins also hardly have the need for using this method. IMO, it would be more meaningful to move up to the interface. Then by this method, we can simply implement for default plugins who actually need args. WDYT?
should remain unexported because it provides pointers. However, I agree that might be too much to ask for all plugins. Instead, we can add a new interface and then plugins can opt-int. Not sure if we should follow this extension pattern Any thoughts
On the other hand, simply returning doesn't seem like much.
Sure, then I will add like what we want before(add this method to interface), and if there is any better idea later, please let me know.
I think we should just add it to the plugins that actually have args, this will require custom code in the test, but I think it is better than having all plugins implement a function means something to only 1% of them.
Thanks, but now there is a problem that we can't get the registered plugins with current API.

So, I'm not sure if we can just implement the method in with some reflect code for checking if one plugin has implement method? This could help us with keeping the interface clean. cc
you don't need to check if it implements BuildArgs, since this is for testing, we can hardcode which plugins should have their args verified.
The test will not be pretty, but I think it is worth the tradeoff.
Do you mean that we don't test the default plugins, but define some custom plugins in the test file? And we don't need the ListPluginArgs method anymore, am I right?
and I discussed this offline and arrived to this conclusion: For the compatiblity test, we can mock ( in ) to capture the arguments the configuration that is being passed. This can also be used by with We want another test for that checks the build args for the in-tree plugins accessing the internal pointers. These 2 tests should cover the whole stack. Sorry for the back-and-forth. Sometimes it's useful to see a full PR to realize that the initial idea wasn't good enough. Thanks a lot for it.
Sure, that sounds much more clear, thanks. :D
One more question, the is under package , and the compatibility test is under package . So for mocking , we need to change to a function variable in , and add a method for it, right It will look like: Maybe we can add something like to , but that's the implementation details. I want to make sure if the above design is proper.
Or rather, you could add something like: . Then you can call the capturer from if there is one defined.
Got your message in slack, let me reply here. sounds good, and do I need to revert the change in ? I think is not necessary now.
I think they will still be useful for the other test:
Hello are you still working on this?
Ah, I'm so sorry that I'm so busy in the last few weeks, but I will send prs fot this issue in this weekend. Sorry for my delay again.


Question: } } <ins> func makeContainersByQOS(class v1.PodQOSClass) []v1.Container { resource := newResourceList("100m", "1Gi", "") switch class { case v1.PodQOSGuaranteed: return []v1.Container{newContainer("guaranteed-container", resource, resource)} case v1.PodQOSBurstable: return []v1.Container{newContainer("burtable-container", resource, nil)} case v1.PodQOSBestEffort: fallthrough default: return []v1.Container{newContainer("best-effort-container", nil, nil)} } } func TestAdmitUnderNodeConditions(t *testing.T) { manager := &managerImpl{} pods := []*v1.Pod{ newPod("guaranteed-pod", scheduling.DefaultPriorityWhenNoDefaultClassExists, makeContainersByQOS(v1.PodQOSGuaranteed), nil), newPod("burstable-pod", scheduling.DefaultPriorityWhenNoDefaultClassExists, makeContainersByQOS(v1.PodQOSBurstable), nil), newPod("best-effort-pod", scheduling.DefaultPriorityWhenNoDefaultClassExists, makeContainersByQOS(v1.PodQOSBestEffort), nil), } expected := []bool{true, true, true} for i, pod := range pods { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Errorf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } manager.nodeConditions = []v1.NodeConditionType{v1.NodeMemoryPressure} expected = []bool{true, true, false} for i, pod := range pods { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Errorf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } manager.nodeConditions = []v1.NodeConditionType{v1.NodeMemoryPressure, v1.NodeDiskPressure} expected = []bool{false, false, false} for i, pod := range pods { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Errorf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } } </ins> // parseQuantity parses the specified value (if provided) otherwise returns 0 value func parseQuantity(value string) resource.Quantity { if len(value) == 0 {

<!-- STOP -- PLEASE READ! GitHub is not the right place for support requests. If you're looking for help, check and the . You can also post your question on the or the forum. If the matter is security related, please disclose it privately via --In the code below, kubelet rejects pods with QOS type guaranteed when under disk pressure, but when kubelet is also under memory pressure, those pods are accepted, is this behavior expected? Why is that?
/area kubelet /sig node
See this comment: It came with commit
you can find an answer from the proposal doc:
thank you ! I think current code is not following the eviction design of condition. the proposal is: but the code is: , guaranteed pods are not rejected any more. seems to be working fine though, maybe here's the reason: PS: above content can be found in


Question: rbac.NewRule("create", "get", "list", "watch").Groups(legacyGroup).Resources("nodes").RuleOrDie(), // TODO: restrict to the bound node once supported rbac.NewRule("update", "patch").Groups(legacyGroup).Resources("nodes/status").RuleOrDie(), <ins> rbac.NewRule("update", "patch", "delete").Groups(legacyGroup).Resources("nodes").RuleOrDie(), </ins> // TODO: restrict to the bound node as creator once supported rbac.NewRule("create", "update", "patch").Groups(legacyGroup).Resources("events").RuleOrDie(),

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubelet RBAC Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE Install tools: make quick-release && go run -up What happened: I upgraded the cluster with After which the master node was not ready. What you expected to happen: I expected the master node to be ready. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: In the api server logs: In the kubelet logs:
cc since you were running into this.
cc
c.f. it seems wrong for the node to delete itself
In the case where a kubelet starts up and sees a mismatch between the externalid field and its own idea of what its external id is, it is trying to delete itself. Questions I have: what can make the external id mismatch occur (in cloudprovider and non-cloudprovider cases)? is it even safe for a kubelet to summarily delete itself (as it is trying to do today), or can that mess up apiserver/controller logic around node cleanup, like the attach/detach controller, etc? can we limit the node to reporting the mismatch by updating node status and stay in the tryRegisterWithApiServer loop, and let something like the node controller remove a node that gets in this situation instead? cc
happens in both GCE and GKE upgrades. who can update node status. I would view this as strictly worse than giving kubelets node delete privileges.
If a node instance is recreated with the same name, it will register as the same node. This apparently also happens in tests (). AFAIK, attach/detach controller annotates the node object to keep states. If the node is backed by a new (VM) instance, it is probably safe/better to recreate the node object and start afresh. Of course, upon any node update/delete/add events, the detach/attach controller should examine and reconcile the status. Hm......AFAIK node controller also does not coordinate with other controllers (e.g., detach/attach). I am not sure how this would help the second point you made.

would you be ok with adding delete permissions for kubelet because it has them and uses them today, and then taking the argument about whether it should be done another way to it's own issue?
yeah, convinced me that if the node is the authority of the spec during self-registration cases, update/delete permissions are reasonable, and we'd need to figure out how to scope those to update/delete self in the future, just like we do for update status
see


Question: func ReadOrDie(filePath string) []byte { data, err := Read(filePath) if err != nil { <del> ginkgo.Fail(err.Error(), 1) </del> <ins> e2elog.Fail(err.Error(), 1) </ins> } return data }

/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign


Question: "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apiserver/pkg/endpoints/metrics:go_default_library", "//staging/src/k8s.io/apiserver/pkg/server/httplog:go_default_library", <del> "//staging/src/k8s.io/client-go/informers:go_default_library", </del> "//vendor/k8s.io/klog/v2:go_default_library", ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: } } <ins> func TestRemoveDuplicateAltNames(t *testing.T) { tests := []struct { args *certutil.AltNames want *certutil.AltNames }{ { &certutil.AltNames{}, &certutil.AltNames{}, }, { &certutil.AltNames{ DNSNames: []string{"a", "a"}, IPs:      []net.IP{{127, 0, 0, 1}}, }, &certutil.AltNames{ DNSNames: []string{"a"}, IPs:      []net.IP{{127, 0, 0, 1}}, }, }, { &certutil.AltNames{ DNSNames: []string{"a"}, IPs:      []net.IP{{127, 0, 0, 1}, {127, 0, 0, 1}}, }, &certutil.AltNames{ DNSNames: []string{"a"}, IPs:      []net.IP{{127, 0, 0, 1}}, }, }, { &certutil.AltNames{ DNSNames: []string{"a", "a"}, IPs:      []net.IP{{127, 0, 0, 1}, {127, 0, 0, 1}}, }, &certutil.AltNames{ DNSNames: []string{"a"}, IPs:      []net.IP{{127, 0, 0, 1}}, }, }, } for _, tt := range tests { RemoveDuplicateAltNames(tt.args) if !reflect.DeepEqual(tt.args, tt.want) { t.Errorf("Wanted %v, got %v", tt.want, tt.args) } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Duplicate altnames in certs which generated by kubeadm ! What you expected to happen: No Duplicate altnames in certs which generated by kubeadm How to reproduce it (as minimally and precisely as possible): Anytime Anything else we need to know?: ! Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/area kubeadm
Can you describe your detailed usage scenarios and reproduction steps, and explain the kubernetes version so that we can quickly locate
Thank for your response! I have found why and will fix it soon.
i have commented on the PR. /priority awaiting-more-evidence
/remove-priority awaiting-more-evidence /priority backlog