Question: export DOCKER_CLI_EXPERIMENTAL := enabled # golang version should match the golang version from https://github.com/coreos/etcd/releases for the current ETCD_VERSION. GOLANG_VERSION?=1.10.4 <del> GOARM=7 </del> <ins> GOARM?=7 </ins> TEMP_DIR:=$(shell mktemp -d) ifeq ($(ARCH),amd64)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: login system root, and run below command from "test/images" dir, Will fail with below error message, ./image- line 91: : command not found This is cause by the empty string like "" cannot be understood as a shell command. What you expected to happen: image should be built successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ):all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: cd $ $ make ../image- bin webhook ../image- line 145: GOARM: unbound variable Makefile:23: recipe for target 'bin' failed make: [bin] Error 1 What you expected to happen: The binary or image should be built successfully. How to reproduce it (as minimally and precisely as possible): see my desc above, type "make" will see the error message. Anything else we need to know?: Environment*: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): Ubuntu Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I'm pretty sure this is supposed to be built via from which does default this environment variable.
, thanks for the information! I think the harness of the shell scripts could bring us a little more, GOARM should only been set when the ARCH is ARM Make a binary is possible where a Makefile is presented
That seems reasonable, but we should probably avoid computing this in many places, currently the GOARM=${GOARM:-7} is going to always be 7 when building in the currently supported way. We should change that if we're going to compute it elsewhere. cc-ing for test/images fun ...


Question: "fmt" "reflect" "regexp" <ins> "strings" </ins> "github.com/emicklei/go-restful/swagger" "github.com/golang/glog"

Using kubectl create with a list no longer works. We get the output:
In kubernetes 1.0 with a List would create all items in the List.
Test data to duplicate the problem:
CC
FYI in case you run into this.
Another feature that needs a test.
the swagger spec may only include object definitions that are registered as API request/response types... since isn't actually submitted to the server (that I know of), that's probably why it didn't get included. Enabling validation by default () probably exposed the bug with a List not being able to validate
We need to make validation on List work. We should test multiple resources w/o List as well.
Naive question: Could we just call Schema() after Flatten() but before Do()?
Hmm. No, validation isn't actually called there.
Each input-source visitor (URLVisitor, FileVisitor, StreamVisitor) has its own call to ValidateSchema. We need to validate before decoding, but we don't unpack the list until after decoding.
We could do something similar to FlattenListVisitor in ValidateSchema: But that's pretty ugly.
Do you have time to take a look at what the most straightforward solution/workaround is for this? Longer-term we should maybe push this into the json decoder, as proposed here:
The simplest thing might be to add List to the swaggerapi models list… I was looking into that yesterday and got sidetracked
I thought about adding List, but it didn't look like the validator would recurse into the individual resource models, hence the need to special-case List.
Nobody is working on this at the moment, if you'd like to take a look.
OK, I will take a look.
If add List to the swaggerapi, I think it need to add a new api . If we post request to to handler the List, need to unpack the list in the server. Or just add an "empty" api only list the List kind ? I think that might be a little strange all above. Please correct me.
I recommended not adding List to the swagger. I don't think it would help. I think we need to detect the case of a List and extract the resources within, similar to FlattenListVisitor.
I agree. I think doing something similar to FlattenListVisitor in ValidateSchema maybe a little ugly. But it's straightforward.


Question: expected := []string{ "Name:s*mypsp", "Allow Privileged:s*false", <ins> "Allow Privilege Escalation:s*false", </ins> "Default Add Capabilities:s*<none>", "Required Drop Capabilities:s*<none>", "Allowed Capabilities:s*<none>",

What happened: When doing , the Allow Privileged Escalation output shows a hex-encoded value instead of a stringified boolean: What you expected to happen: Show the stringified boolean value instead. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Solution turned out to be simple. Change this line: to be instead of .
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This was fixed with


Question: description: "multiple sandboxes should be handled properly.", sandboxes: []sandboxTemplate{ // running sandbox. <del> makeGCSandbox(pods[0], 1, runtimeapi.PodSandboxState_SANDBOX_READY, true), </del> <ins> makeGCSandbox(pods[0], 1, runtimeapi.PodSandboxState_SANDBOX_READY, true, 1), </ins> // exited sandbox without containers. <del> makeGCSandbox(pods[0], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false), </del> <ins> makeGCSandbox(pods[0], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true, 0), </ins> // exited sandbox with containers. <del> makeGCSandbox(pods[1], 1, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true), </del> <ins> makeGCSandbox(pods[1], 1, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true, 1), </ins> // exited sandbox without containers. <del> makeGCSandbox(pods[1], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false), </del> <ins> makeGCSandbox(pods[1], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, true, 0), </ins> // exited sandbox without containers for deleted pods. <del> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false), </del> <ins> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false, 0), </ins> }, containers: []containerTemplate{ {pod: pods[1], container: &pods[1].Spec.Containers[0], sandboxAttempt: 1, state: runtimeapi.ContainerState_CONTAINER_EXITED},

kubernetes pre-submit unit tests for pass some times and fails some times causing the flakiness. Here is one such failure https://k8s- Passing: Failing: cc
Yep, I'm checking with this
Sorry for introducing a flakiness. Sent out the fix in .
/assign
Dup of . /close

https://k8s- Failed: TestSandboxGC {kuberuntime}
https://k8s- Failed: TestSandboxGC {kuberuntime}
/assign
Should've been fixed by


Question: return nil } <del> // appendListItem decodes and appends the object (if it passes filter) to v, which must be a slice. func appendListItem(v reflect.Value, data []byte, rev uint64, pred storage.SelectionPredicate, codec runtime.Codec, versioner storage.Versioner, newItemFunc func() runtime.Object) error { </del> <ins> // decodeListItem decodes bytes value in array into object. func decodeListItem(ctx context.Context, data []byte, rev uint64, codec runtime.Codec, versioner storage.Versioner, newItemFunc func() runtime.Object) (runtime.Object, error) { startedAt := time.Now() defer func() { endpointsrequest.TrackDecodeLatency(ctx, time.Since(startedAt)) }() </ins> obj, _, err := codec.Decode(data, nil, newItemFunc()) if err != nil { <del> return err </del> <ins> return nil, err </ins> } <del> // being unable to set the version does not prevent the object from being extracted </del> if err := versioner.UpdateObject(obj, rev); err != nil { klog.Errorf("failed to update object version: %v", err) } <del> if matched, err := pred.Matches(obj); err == nil && matched { v.Set(reflect.Append(v, reflect.ValueOf(obj).Elem())) } return nil </del> <ins> return obj, nil </ins> } // recordDecodeError record decode error split by object type.

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: staticpodutil "k8s.io/kubernetes/cmd/kubeadm/app/util/staticpod" authzmodes "k8s.io/kubernetes/pkg/kubeapiserver/authorizer/modes" "k8s.io/kubernetes/pkg/master/reconcilers" <ins> utilpointer "k8s.io/kubernetes/pkg/util/pointer" </ins> "k8s.io/kubernetes/pkg/util/version" )

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: clearExpectations(t, manager, ds, podControl) } <ins> func TestDaemonSetUpdatesSomeOldPodsNotReady(t *testing.T) { _, ctx := ktesting.NewTestContext(t) ds := newDaemonSet("foo") manager, podControl, _, err := newTestController(ctx, ds) if err != nil { t.Fatalf("error creating DaemonSets controller: %v", err) } maxUnavailable := 2 addNodes(manager.nodeStore, 0, 5, nil) err = manager.dsStore.Add(ds) if err != nil { t.Fatal(err) } expectSyncDaemonSets(t, manager, ds, podControl, 5, 0, 0) markPodsReady(podControl.podStore) ds.Spec.Template.Spec.Containers[0].Image = "foo2/bar2" ds.Spec.UpdateStrategy.Type = apps.RollingUpdateDaemonSetStrategyType intStr := intstr.FromInt(maxUnavailable) ds.Spec.UpdateStrategy.RollingUpdate = &apps.RollingUpdateDaemonSet{MaxUnavailable: &intStr} err = manager.dsStore.Update(ds) if err != nil { t.Fatal(err) } // All old pods are available, should update 2 clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, maxUnavailable, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, maxUnavailable, 0, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, 0, 0) clearExpectations(t, manager, ds, podControl) // Perform another update, verify we delete and create 2 pods, three available should remain untouched ds.Spec.Template.Spec.Containers[0].Image = "foo2/bar3" err = manager.dsStore.Update(ds) if err != nil { t.Fatal(err) } clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, maxUnavailable, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, maxUnavailable, 0, 0) clearExpectations(t, manager, ds, podControl) expectSyncDaemonSets(t, manager, ds, podControl, 0, 0, 0) clearExpectations(t, manager, ds, podControl) readyCount := 0 expectedReadyCount := 5 - maxUnavailable for _, obj := range podControl.podStore.List() { pod := obj.(*v1.Pod) n, condition := podutil.GetPodCondition(&pod.Status, v1.PodReady) if n != -1 && condition.Status == v1.ConditionTrue { readyCount++ } } if readyCount != expectedReadyCount { t.Fatalf("Expected %d old ready pods, but found %d", expectedReadyCount, readyCount) } } </ins> func TestDaemonSetUpdatesAllOldPodsNotReady(t *testing.T) { _, ctx := ktesting.NewTestContext(t) ds := newDaemonSet("foo")

Starting with a cluster of three nodes, with a healthy deamonset pod running on each, with . When performing three updates to the daemonset, each with some reason that prevents the new pods from becoming ready, I ended up with zero ready pods. I would expect that when I apply an update, and the new pods fail to become ready, when I apply a second update the "new" failing pods are replaced before considering any of the oldest healthy pods. This is similar behavior to how Deployments work. On a three node (or any count really) cluster, create this daemonset, which is just an nginx pod with a startup probe hitting port 80. Update the startup probe port to something other than port 80, which should cause the pod to never become ready, and apply again. You can see the new unhealthy pod, and the generation of each pod Repeat this process for as many nodes/pods as you have (2 more in this example), changing anything that would cause the pods to be recreated (in this case I just cycled the startup probe port a few times). After the 2nd update, you can see that it has replaced both the 2nd generation pod (the one that was failing), and one of the first generation pods. Im now down to 1 out of 3, when I had specified . One more apply / update, and all generation 1 pods are gone, and all pods are unhealthy. No response <details</details<detailsReproduced this on bare metal, and GKE/GCP </detailsNo response No response No response No response
/sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/triage accepted /priority important-longterm


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/conversion" ) <ins> func IsListType(obj Object) bool { _, err := GetItemsPtr(obj) return err == nil } </ins> // GetItemsPtr returns a pointer to the list object's Items member. // If 'list' doesn't have an Items member, it's not really a list type // and an error will be returned.

Had a report of this over email. Filing this issue so we don't forget to investigate.
Can't reproduce.


Question: func TestServiceRegistryMakeLinkVariables(t *testing.T) { service := api.Service{ <del> JSONBase:      api.JSONBase{ID: "foo"}, </del> <ins> JSONBase:      api.JSONBase{ID: "foo-bar"}, </ins> Selector:      map[string]string{"bar": "baz"}, ContainerPort: util.IntOrString{Kind: util.IntstrString, StrVal: "a-b-c"}, } <del> vars := makeLinkVariables(service, "mars") </del> <ins> registry := registrytest.NewServiceRegistry() registry.List = api.ServiceList{ Items: []api.Service{service}, } machine := "machine" vars, err := GetServiceEnvironmentVariables(registry, machine) if err != nil { t.Errorf("unexpected err: %v", err) } </ins> for _, v := range vars { if !util.IsCIdentifier(v.Name) { t.Errorf("Environment variable name is not valid: %v", v.Name)

In issue it appeared that consensus was reached that a service id could contain a hyphen, but that it would be translated to an underscore in the environment variables (this issues was the root problem, back in ). However, the replacement does not appear to have been done: hyphens are still not replaced with underscores in env-var names. (Although now underscores are blocked in service names!)
Hm, that's weird. I thought that this code was doing replacement. Did that get removed?
Thanks for looking into this lavalamp. Was your question directed at me? I don't see any string replacement code in the code you linked to, at least not for the (I do see escaping for the port in line 198)
Ah, no, I was just thinking out loud there. Maybe knows what happened?
Ah, that's an oversight. We do it for the Docker link style variables, but not the old school ones. I'll send a PR.
PR sent.


Question: By("test for subpath mounted with old value") cmd := "test -f /volume_mount/foo/test.log" <del> _, err = framework.RunHostCmd(pod.Namespace, pod.Name, cmd) </del> <ins> _, _, err = f.ExecShellInPodWithFullOutput(pod.Name, cmd) </ins> if err != nil { framework.Failf("expected to be able to verify old file exists") } cmd = "test ! -f /volume_mount/newsubpath/test.log" <del> _, err = framework.RunHostCmd(pod.Namespace, pod.Name, cmd) </del> <ins> _, _, err = f.ExecShellInPodWithFullOutput(pod.Name, cmd) </ins> if err != nil { framework.Failf("expected to be able to verify new file does not exist") }

Which jobs are failing: Tests which exec into containers and alter lifecycles Which test(s) are failing: All new tests introduced for api change which exec into containers New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Since when has it been failing: 27/February/2019 Testgrid link: https://k8s- Reason for failure: is only set in e2e tests and the binary is not available in node e2e tests Need to restructure and call the API rather than through the framework convenience function Anything else we need to know:
/sig node /sig storage /assign /cc


Question: // read or write a certificate stored/embedded in a file type certificateReadWriter interface { //Exists return true if the certificate exists <del> Exists() bool </del> <ins> Exists() (bool, error) </ins> // Read a certificate stored/embedded in a file Read() (*x509.Certificate, error)

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: func waitForInformerSync(ctx context.Context, t *testing.T, informer cache.SharedIndexInformer, wantSynced bool, lastSyncResourceVersion string) { t.Helper() <del> syncCtx, cancel := context.WithTimeout(ctx, time.Second*30) </del> <ins> syncCtx, cancel := context.WithTimeout(ctx, time.Second*60) </ins> defer cancel() if gotSynced := cache.WaitForCacheSync(syncCtx.Done(), informer.HasSynced); wantSynced != gotSynced { t.Fatalf("wanted sync %t, got sync %t", wantSynced, gotSynced) } if len(lastSyncResourceVersion) != 0 { <del> if err := wait.PollImmediate(time.Second, time.Second*30, func() (bool, error) { </del> <ins> if err := wait.PollImmediate(time.Second, time.Second*60, func() (bool, error) { </ins> return informer.LastSyncResourceVersion() != lastSyncResourceVersion, nil }); err != nil { t.Fatalf("informer never changed resource versions from %q: %v", lastSyncResourceVersion, err)

wanted sync true, got sync false wanted sync true, got sync false wanted sync true, got sync false informer never changed resource versions from "": timed out waiting for the condition /kind failing-test /kind flake as it is in /sig auth Open it as it is the top flake of pull-kubernetes-integration:
Commented on PR.
/triage accepted
I've seen this test fail 2 times today on my PR Going to hope that I close my eyes and and never see it if it happens again I'll reopen this...
!
Opened to track flakes once again


Question: } // The bound volume type may change invalidPredicates.Insert(maxPDVolumeCountPredicateKeys...) <del> // The bound volume's label may change invalidPredicates.Insert("NoVolumeZoneConflict") </del> } c.equivalencePodCache.InvalidateCachedPredicateItemOfAllNodes(invalidPredicates)

What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate).

is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.


Question: const ( dockerConfigPrefix           = "DOCKER_" <del> latestValidatedDockerVersion = "18.09" </del> <ins> latestValidatedDockerVersion = "19.03" </ins> ) // Validate is part of the system.Validator interface.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Support for newer docker versions: 19.x Why is this needed: New users trying to use kubernetes install the latest docker version. It's inconvenient see the following warning and have to uninstall docker and then install an older version. Also, it's not clear what could happen if someone uses a newer version of docker. Lastly, I believe that kubernetes should be able to keep up with docker. Is there any roadmap for this, that I have missed?
/sig usability
you could work with sig-testing and sig-node to set up CI jobs when new versions of Docker shows up. Believe me, it's more inconvenient to debug failures in a new docker version as a end use than the downgrade. We have to have some CI coverage of a version before we ok its use by the community.
Hi I'm new to this, so I'm not sure how to apply your suggestion: Is there any documentation that I could use?
SIGs are teams working on specific areas of kubernetes All our test infrastructure for CI jobs ("Continuous Integration") is in: Out test-grid is here (please type docker in the search box on top to see the current CI jobs):
Since docker-ce 18.09 has reached EOL and there are no new versions planned (), it would be great to have official support for a newer version of docker-ce.
please see:
Hi folks, David from Docker here, we have run and validated Docker Engine 19.03 for Kubernetes 1.14 back in June, so Docker 19.03 is indeed supported for Kubernetes 1.14 and above: Let me know how else we can help.
thanks for confirming PR is here
Thank you Appreciate the PR, we typically test engines for major versions of Engine that are released like 19.03 with the latest version of Kubernetes that is available at the time. On our next engine, we will re-run the test and submit the results to testgrid. Let me know how I can better coordinate this.
Also let me know if there is anything else that is needed to close this issue out. Thanks.
hi, the PR should close this issue. i guess, one option is just creating a similar issue once the test job is available in test-grid and running green for a while? or me can then proceed to update the validators.

Ok thank you, that sounds like a good process we can follow moving forward.
Hi Any chance this can make its way into 1.16 at least? This way it may get picker up by Kops and not have to rely on some old and unpatched Docker version. At the moment Docker CE 18.09 is no longer actively supported
hi, we cannot backport this as it's not a critical fix. at least in the case of kubeadm the 19. versions of docker would only print a warning.
but doesn't the existence of a vulnerability make this critical enough?
the usage of Docker 19. is not blocked in any way in 1.16. you can ignore the and just use it. if something was blocking the usage of 19., then this is backport worthy.
I saw this as the first step before adding 19.03 as a validated external release. Without being documented as such Kops will not pick it up as default and will leave people with old and vulnerable Docker versions. If this I understand correctly the above comments, 19.03 has been validated for some time, just not documented.
i don't think kops uses the kubeadm validators that reside in k/k/cmd/kubeadm/app/... this was resolved in this PR the k8s docs for CRI are still out of date: so if someone wants to send a PR for that page please go ahead: (but AFAIK not all distros have moved to 19. yet)
I couldn't find any repos missing docker 19.03.4. Any chance to do the change in the changelog?


Question: if err := rest.BeforeUpdate(e.UpdateStrategy, ctx, obj, existing); err != nil { return nil, nil, err } <ins> // Ignore changes that only affect managed fields timestamps. // FieldManager can't know about changes like normalized fields, defaulted // fields and other mutations. obj, err = fieldmanager.IgnoreManagedFieldsTimestampsTransformer(ctx, obj, existing) if err != nil { return nil, nil, err } </ins> // at this point we have a fully formed object.  It is time to call the validators that the apiserver // handling chain wants to enforce. if updateValidation != nil {

Sending a patch built from a typed client to SSA will force a write to etcd (and bump resourceVersion) since creationTimestamp is serialized to null. This is a follow-up of which doesn't solve the issue properly. Sending a no-op SSA request with creationTimestamp to null shouldn't do anything Kubectl apply an object with creationTimestamp null twice and see that it updates the object. Probably something like this would cause it:
/sig api-machinery /wg api-expression /triage accepted /assign /priority important-soon


Question: subsets                  []corev1.EndpointSubset epLabels                 map[string]string epAnnotations            map[string]string <ins> uid                      string </ins> endpointsDeletionPending bool maxEndpointsPerSubset    int32 existingEndpointSlices   []*discovery.EndpointSlice

create a service without selector, and manual create endpoints for this service endpointslicemirroring controller will create an endpointslice for this service. after endpoint slice created, and kube-controller-manager restart. during the kube-controller-manager restart, the endpoint was deleted and recreated after kube-controller-manager restart, garbagecollector controller delete this endpointslice and endpointslice not recreate until endpoint/service update event or kube-controller-manager restart endpointslice will be recreated create a service without selector, and manual create endpoints for this service endpointslicemirroring controller will create an endpointslice for this service. after endpoint slice created, and kube-controller-manager restart. during the kube-controller-manager restart, the endpoint was deleted and recreated No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/assign
Trying to understand this, mostly just restating what you've said above to make sure I'm getting it: during the kube-controller-manager restart, the endpoint was deleted and recreated after kube-controller-manager restart, garbagecollector controller delete this endpointslice and endpointslice not recreate until endpoint/service update event or kube-controller-manager restart So an Endpoints resource is being deleted while kube-controller-manager is not running. The EndpointSlice mirroring controller to point to the Endpoints they're mirroring. When kube-controller-manager is restarted, the garbage collector will delete the mirrored EndpointSlice because it's owned by an Endpoints resource that was previously deleted. That deletion by garbage collection would then be , but if the EndpointSliceTracker doesn't think it should have that resource, it will ignore the deletion and not reconcile, resulting in the mirrored EndpointSlice not being recreated until something updates the Service or Endpoints resource.
/triage accepted


Question: stopCh := make(chan struct{}) defer close(stopCh) go gc.Run(1, stopCh) <del> go gc.Sync(fakeDiscoveryClient, 10*time.Millisecond, stopCh) </del> <ins> // The pseudo-code of GarbageCollector.Sync(): // GarbageCollector.Sync(client, period, stopCh): //    wait.Until() loops with `period` until the `stopCh` is closed : //        wait.PollImmediateUntil() loops with 100ms (hardcode) util the `stopCh` is closed: //            GetDeletableResources() //            gc.resyncMonitors() //            controller.WaitForCacheSync() loops with `syncedPollPeriod` (hardcoded to 100ms), until either its stop channel is closed after `period`, or all caches synced. // // Setting the period to 200ms allows the WaitForCacheSync() to check for cache sync ~2 times in every wait.PollImmediateUntil() loop. go gc.Sync(fakeDiscoveryClient, 200*time.Millisecond, stopCh) </ins> // Wait until the sync discovers the initial resources fmt.Printf("Test output")

Users of the function would expect the close of the "done" channel should terminate the . We should change the function to align to users' expectation. We should document what the channel does. This is a behavior change, to be cautious, we should add a release note claiming breaking change. /kind bug
/sig api-machinery cc Feel free to assign to yourself if you have time. Otherwise I'll attempt a fix tomorrow ;)
I'll fix it. /assign
GitHub didn't allow me to assign the following users: kdada. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : I'll fix it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: Scaling foo-v2 up to 20 Scaling foo-v1 down to 0 `, <ins> }, { name:        "1->1 0/1 scale down unavailable rc to a ready rc (rollback)", oldRc:       oldRc(1, 1), newRc:       newRc(1, 1), newRcExists: true, maxUnavail:  intstr.FromInt(0), maxSurge:    intstr.FromInt(1), expected: []interface{}{ up{1}, down{oldReady: 0, newReady: 1, to: 0}, }, output: `Continuing update with existing controller foo-v2. Scaling up foo-v2 from 1 to 1, scaling down foo-v1 from 1 to 0 (keep 1 pods available, don't exceed 2 pods) Scaling foo-v1 down to 0 `, </ins> }, }

I intentionally cause the rolling update to fail by specifying a non-existent image in the spec, so that it times out. rollback then fails with a timeout as well, despite the original replication controller having 1 healthy replica: This is on 1.1.2. I expect the rollback to succeed as the original rc (copper) has 1 healthy replica still.
With 2 replicas the rollback seems to work as expected.
I found the bug inside of the function of . It erroneously assumes reducing replicas on scale down will always drop the minimum available, when this isn't the case if you are rolling back from a failed deployment (where the scale down will be on the NotReady/failed pods). It needs to consider not ready replicas as well in each RC when deciding if it can scale down. This test case illustrates the problem:


Question: "golang.org/x/net/html/atom" "k8s.io/klog/v2" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/sets" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I upgraded from 1.15 and am using prometheus to scrape my pods via the proxy functionality. Previously no error message was being printed, now with 1.19.9 every try results in an error: Example url: https://x.x.x.x:6443/api/v1/namespaces/staging/pods/proxy-app--6sxfx:80/proxy/metrics What you expected to happen: No error message to be logged, since the apiserver anyways returns a corresponding http error code. The least thing I would expect it not to log as an error, but as a debug or info log, since it doesn't actually represent an error. It returns something like that: How to reproduce it (as minimally and precisely as possible): Call the proxy functionality on an endpoint that does not listen and will refuse the connection. Anything else we need to know?: There is a lot of other issues with a similar error message, but I believe they are not interconnected since this has explicitly todo with an error raised from the proxy functionality. Environment: Kubernetes version (use ): 1.19.9 Cloud provider or hardware configuration: on premise OS (e.g: ): CentOs 7 Kernel (e.g. ): 5.4.60
/sig api-machinery
/cc /triage accepted
/assign


Question: "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//vendor/k8s.io/client-go/util/testing:go_default_library", ],

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: "fmt" "io" "math" <del> "os/exec" "strings" </del> "time" dockertypes "github.com/docker/docker/api/types" <del> "k8s.io/klog" </del> "k8s.io/client-go/tools/remotecommand" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/runtime/v1alpha2" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.


Question: Runonce                 bool } <del> func createAndInitKubelet(kc *KubeletConfig, pc *config.PodConfig) *kubelet.Kubelet { </del> <ins> func createAndInitKubelet(kc *KubeletConfig, pc *config.PodConfig) (*kubelet.Kubelet, error) { </ins> // TODO: block until all sources have delivered at least one update to the channel, or break the sync loop // up into "per source" synchronizations <del> k := kubelet.NewMainKubelet( </del> <ins> k, err := kubelet.NewMainKubelet( </ins> kc.Hostname, kc.DockerClient, kc.EtcdClient,

Things like SyncFrequency should have a lower-bound of acceptable, and that really should be 0ns.
The main point of this is now fixed, though we could be better about input checking all over


Question: numNodes, err := e2enode.TotalRegistered(f.ClientSet) framework.ExpectNoError(err) replicas := int32(numNodes) <del> common.NewRCByName(c, ns, name, replicas, &gracePeriod) </del> <ins> common.NewRCByName(c, ns, name, replicas, &gracePeriod, []string{"serve-hostname"}) </ins> err = e2epod.VerifyPods(c, ns, name, true, replicas) framework.ExpectNoError(err, "Each pod should start running and responding")

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes Since when has it been failing: These tests have been failing since 6/28 Failures coincide with the following changes: Testgrid link: Reason for failure: The failure messages look as follows: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes /cc /milestone v1.16 /priority critical-urgent /kind failing-test /cc
/sig cluster-lifecycle
For more context, the code for the failing tests is here The error for both comes through  - some Pod is not responding. It is failing to find pods prefixed with "my-hostname-add-node". Within the logs we can see this: So the Pods are stuck in a Pending state. The Pods are supposed to be created by a ReplicationController in , which itself uses this method from the e2e framework . However, after being in a Pending state (after the "failed to wait for pods responding: timed out waiting for the condition" error is registered), they do seem to reach a Ready state
/assign for triage.
I think agnhost needs to be passed , but did not do so in /assign sig-testing
GitHub didn't allow me to assign the following users: sig-testing. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I think agnhost needs to be passed , but did not do so in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopen to monitor job /reopen
Reopened this issue.

<detailsIn response to : Reopen to monitor job /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for working on the PR to solve this. indeed solved the issue reported here. /close
Closing this issue. <detailsIn response to : Thank you for working on the PR to solve this. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: for _, group := range groups { if !s.isGroupValid(group) { <del> detail := fmt.Sprintf("%d is not an allowed group", group) </del> <ins> detail := fmt.Sprintf("group %d must be in the ranges: %v", group, s.ranges) </ins> allErrs = append(allErrs, field.Invalid(field.NewPath(s.field), groups, detail)) } }

Is this a BUG REPORT or FEATURE REQUEST?: This is request for improvements. What happened: When PodSecurityPolicy provider validates or fields of a pod, it shows an error like "999 is not an allowed group" if the group doesn't belong to the range(s). What you expected to happen: It should include to the error message also the allowed values -- the ranges against what we were validating this group. This will simplify user experience and debugging. How to reproduce it (as minimally and precisely as possible): run kubernetes cluster with PSP support create a PSP that allows fsGroup/SupplementalGroups in some ranges grant to a user access to this PSP try to create a pod that has fsGroup/SupplementalGroups outside of the allowed ranges Anything else we need to know?: In order to improve the current behavior, we need to modify function here: Reported-by: /kind help
This looks like a good first issue! My suggestion would be something like . I would also suggest to merge overlaying ranges in the log message, like as , to keep the log as concise as possible. I will look into it.
Maybe something like ? Thank you!
We can use the same approach that strategy uses:
Sure, that is probably the most consistent solution.
/remove-help


Question: assert.True(t, apiequality.Semantic.DeepEqual(testCase.expectedAddresses, existingNode.Status.Addresses), "Diff: %s", diff.ObjectDiff(testCase.expectedAddresses, existingNode.Status.Addresses)) <ins> if testCase.expectedAnnotations != nil { assert.True(t, apiequality.Semantic.DeepEqual(testCase.expectedAnnotations, existingNode.Annotations), "Diff: %s", diff.ObjectDiff(testCase.expectedAnnotations, existingNode.Annotations)) } </ins> }) } }

I am reproducing this in OpenShift using the OpenStack cloud provider, but nothing here is specific to either OpenShift or OpenStack. When migrating a cluster which uses in-tree cloud provider to use external cloud-provider, we use the following process: cloud controllers in KCM CCM kubelets with --cloud-provider=external Between steps 1&2 the node-lifecycle, routes, and service controllers are not running anywhere, but this situation isn't unsafe for the cluster and we can keep it short. Between steps 2&3 is more interesting. The node controller is running in CCM, and the kubelets are running with --cloud-provider=openstack (or whatever, just not external). The main loop of node-controller will ignore a newly created node because . However, the loop does not have a similar exclusion. . Note that we shortcut if the kubelet is using an external cloud provider, but not if we don't. So between steps 2&3 both kubelet and CCM's node controller are managing Node.Addresses. If the legacy cloud provider and external CCM don't return the same list of NodeAddresses this causes flapping. Running OpenShift on OpenStack we have 2 problems. Firstly, our legacy cloud provider and external CCM produce different lists. We don't think this would be a problem in practise, except for the second problem. Kubelet adds the annotation to the node containing the node IP. This allows the CCM node controller to apply the same filtering based on node IP as kubelet does for the legacy cloud provider. This means that, although the lists are different, the node IP is still listed first which is the only thing that matters in practise. The problem is that kubelet only adds the annotation when --cloud-provider=external which, between steps 2&3, it is not. This means that Node.Addresses flaps between a correct and a broken configuration, as the NodeAddresses returned by the external CCM cannot be filtered by node IP without the annotation. The ideal solution here would be to ensure that only 1 controller is managing Node.Addresses during this period in the same way we do for Node creation with the external CCM taint. This would fix the issue entirely.

A fix with a much smaller scope would be to have kubelet add the node IP annotation unconditionally so, although Node.Address can potentially flap, it will at least be between 2 valid configurations. Given that this should only ever be a problem during the upgrade window, the smaller fix may be more expedient. Node.Addresses should remain valid at all times when kubelet is configured with --cloud-provider=openstack but the CCM is also running. Good luck No response 24 OpenStack (but as described, not OpenStack-specific) No response No response No response No response
/sig cloud-provider
we've an agenda item for next week's (11 May) SIG cloud provider meeting about this topic
/triage accepted
Would it work to swap steps 2 and 3?
Good question. I don't think so. That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean: No cluster scale-up or scale-down No creation, deletion, or modification to LB Services I think we could handle this in OpenShift, although it's a bit fragile and I suspect updating all the kubelets can take a really long time on a big cluster. However, I think this would break cluster-api or any deployment tooling which rolls out updates by creating new machines. I think this would never complete: the new nodes would never come up because the node controller isn't running.
I am still trying to understand the issues here while reviewing your pr: , but I agreed with your comments above: That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean:


Question: // case 8 { devAddresses: map[string][]string{"kube-ipvs0": {"1.2.3.4", "2.3.4.5"}, "eth5": {"3.4.5.6"}, "lo": {"127.0.0.1"}}, <del> expectIPs:    []string{"127.0.0.1", "3.4.5.6"}, </del> <ins> expectIPs:    []string{"3.4.5.6"}, </ins> }, // case 9 { devAddresses: map[string][]string{"ipvs0": {"1.2.3.4"}, "lo": {"127.0.0.1"}, "encap7": {"10.20.30.31"}}, <del> expectIPs:    []string{"127.0.0.1", "10.20.30.31", "1.2.3.4"}, </del> <ins> expectIPs:    []string{"10.20.30.31", "1.2.3.4"}, </ins> }, }

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: if err != nil { return nil, err } <ins> serviceTags, err := getServiceTags(service) if err != nil { return nil, err } </ins> var sourceAddressPrefixes []string <del> if sourceRanges == nil || serviceapi.IsAllowAll(sourceRanges) { </del> <ins> if (sourceRanges == nil || serviceapi.IsAllowAll(sourceRanges)) && len(serviceTags) == 0 { </ins> if !requiresInternalLoadBalancer(service) { sourceAddressPrefixes = []string{"Internet"} }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When using the loadBalancerSourceRanges property of a service in a manifest, we are force to use a CIDR block. This was discovered while debugging the following issue. What you expected to happen: Azure allows us to use Service Tags (e.g. VirtualNetwork, Internet, etc) WE should be able to specify them form a k8s service manifest. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.11 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): centOS 7.3 Kernel (e.g. ): 3.10 Install tools: Others:
/sig azure
Here's the requested Feature.
SourceRanges is part of service spec, while serviceTag is azure specific. I think we don't expect to add cloud-specific features into Kubernetes API. What about adding a new annotation for this? e.g.
I believe that at this stage, (no out of tree cloud provider logic), using an annotation would be the only way to go unfortunately.
/priority important-soon
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // maxTries == minTries will confirm that we see the expected endpoints and no // more for maxTries. Use this if you want to eg: fail a readiness check on a // pod and confirm it doesn't show up as an endpoint. <del> func (config *NetworkingTestConfig) DialFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort, maxTries, minTries int, expectedResponses sets.String) { </del> <ins> // Returns nil if no error, or error message if failed after trying maxTries. func (config *NetworkingTestConfig) DialFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort, maxTries, minTries int, expectedResponses sets.String) error { </ins> ipPort := net.JoinHostPort(containerIP, strconv.Itoa(containerHTTPPort)) cmd := makeCURLDialCommand(ipPort, dialCommand, protocol, targetIP, targetPort)

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: }) }) <ins> ginkgo.It("should enforce except clause while egress access to server in CIDR block [Feature:NetworkPolicy]", func() { protocolUDP := v1.ProtocolUDP // Getting podServer's status to get podServer's IP, to create the CIDR with except clause podServerStatus, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), podServer.Name, metav1.GetOptions{}) if err != nil { framework.ExpectNoError(err, "Error occurred while getting pod status.") } podServerAllowCIDR := fmt.Sprintf("%s/24", podServerStatus.Status.PodIP) // Exclude podServer's IP with an Except clause podServerExceptList := []string{fmt.Sprintf("%s/32", podServerStatus.Status.PodIP)} // client-a can connect to server prior to applying the NetworkPolicy ginkgo.By("Creating client-a which should be able to contact the server.", func() { testCanConnect(f, f.Namespace, "client-a", service, 80) }) policyAllowCIDRWithExcept := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      "deny-client-a-via-except-cidr-egress-rule", }, Spec: networkingv1.NetworkPolicySpec{ // Apply this policy to the client. PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{ "pod-name": "client-a", }, }, PolicyTypes: []networkingv1.PolicyType{networkingv1.PolicyTypeEgress}, // Allow traffic to only one CIDR block except subnet which includes Server. Egress: []networkingv1.NetworkPolicyEgressRule{ { Ports: []networkingv1.NetworkPolicyPort{ // Allow DNS look-ups { Protocol: &protocolUDP, Port:     &intstr.IntOrString{Type: intstr.Int, IntVal: 53}, }, }, }, { To: []networkingv1.NetworkPolicyPeer{ { IPBlock: &networkingv1.IPBlock{ CIDR:   podServerAllowCIDR, Except: podServerExceptList, }, }, }, }, }, }, } policyAllowCIDRWithExcept, err = f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(context.TODO(), policyAllowCIDRWithExcept, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error occurred while creating policy: policyAllowCIDRWithExcept.") defer cleanupNetworkPolicy(f, policyAllowCIDRWithExcept) ginkgo.By("Creating client-a which should no longer be able to contact the server.", func() { testCannotConnect(f, f.Namespace, "client-a", service, 80) }) }) </ins> ginkgo.It("should enforce policies to check ingress and egress policies can be controlled independently based on PodSelector [Feature:NetworkPolicy]", func() { var serviceA, serviceB *v1.Service var podA, podB *v1.Pod

What happened: A missing test case was found for NetworkPolicy with Except clause What you expected to happen: Expect Except clause test to be covered
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: serviceAccounts := e.client.Core().ServiceAccounts(serviceAccount.Namespace) liveServiceAccount, err := serviceAccounts.Get(serviceAccount.Name, metav1.GetOptions{}) if err != nil { <del> // Retry for any error other than a NotFound return !apierrors.IsNotFound(err), err </del> <ins> // Retry if we cannot fetch the live service account (for a NotFound error, either the live lookup or our cache are stale) return true, err </ins> } if liveServiceAccount.ResourceVersion != serviceAccount.ResourceVersion { <del> // our view of the service account is not up to date // we'll get notified of an update event later and get to try again glog.V(2).Infof("serviceaccount %s/%s is not up to date, skipping token creation", serviceAccount.Namespace, serviceAccount.Name) return false, nil </del> <ins> // Retry if our liveServiceAccount doesn't match our cache's resourceVersion (either the live lookup or our cache are stale) glog.V(4).Infof("liveServiceAccount.ResourceVersion (%s) does not match cache (%s), retrying", liveServiceAccount.ResourceVersion, serviceAccount.ResourceVersion) return true, nil </ins> } // Build the secret

At least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created. (The service account itself is, but not the default secret.) Namespace creation + apiserver logs logstash-ingester: namespace creation triggered service account creation but no secret logstash-broker: namespace creation triggered service account creation and secret creation Resulting service accounts Resulting secrets Controller-manager logs for broken case Controller-manager logs for working case Version Info:
Dupe of , fixed in
Actually, not a dupe. This may be a consequence of other refactors to controller logic - possibly we are missing / dropping events out of the queue. I would expect it to get fixed on the sync interval though.
We are also seeing this happening from time to time. In our case, it wasn't fixed in the next sync interval but it took actually a little bit more of 24 hours.
can we resolve this and remove it from the 1.6 milestone?
fixed in in 1.4
We just hit this exact issue running 1.5.2. Maybe the fix didn't get the root cause or there was a regression since 1.4? This should be reopened. I waited about an hour to see if the sync interval would resolve this but it did not. I fixed by deleting and recreating the namespace. I can share logs, but they are identical to what the OP noted. Version info:
Working with this seems like it's not fixed in v1.5.2 -- what should we do here? We don't have permission to re-open the ticket, but filing the same ticket with the same logs (just fresher dates) seems passive-aggressive. :-) What's your preference?
no problem, just getting back to this. can reopen this issue.
can you recreate with the controller manager running at and include controller manager log lines from ?
Fresh repo, offending namespace "foo96", with "--v=5" since we already have that logging level in place on my dev stack. This took 96 repros of a "create namespace / monitor for token creation / delete namespace" loop.

Our script's commands, starting just after 16:36:43: Controller manager log lines from The first reference to the offending namespace (foo96) and the one other entry afterwards: All controller manager log lines citing the offending namespace: For comparison, the previous successfully-created namespace: Status as of 20 minutes later:
Also, here's the apiserver logs + controller manager logs, for more action/reaction context: Namespace: foo96 (problem) Namespace: foo95 (success)
that's helpful, thanks. it looks like while the token controller is processing the creation event for the service account, it does a live lookup against etcd and fails to find the service account. what version of etcd are you running with? are you running HA etcd? do you have quorum read enabled?
We run etcd version 2.3.7 (previously saw this on 2.2.x as well). We run etcd HA, with a 3 node cluster. We do not run the apiserver with the --etcd-quorum-read flag enabled.
(Note: My specific logs above from this morning are on etcd 2.2.5, but yeah we've seen repro on 2.3.7 as well.)
yeah, the non-quorum HA etcd config is the issue. I can look into updating the token controller to tolerate that, but I'm positive there are similar issues in many controllers that assume live lookups are reliable
Ahh excellent, glad to have an explanation! So -- now that we know what to google for, we found -- where you and clayton addressed what appears to be this same problem for service accounts, and specifically noted that you wanted to avoid recommending quorum reads. Why is the answer different here? Is it because the downstream impact of non-quorum reads is too diverse to reliably retry every relevant action? (Totally fine if that's the answer, just trying to make sure we understand the reason for what appears to be an entirely opposite conclusion to the same type of problem.)

I still think controllers should be built to tolerate weak reads, and I'm willing to fix the token controller bug here, but three factors make me think this position is a losing battle: quorum read got a lot cheaper with etcd3 there are strong opinions that etcd quorum read should be on () which means that very few people are writing controllers designed to tolerate weak reads all the CI tests I'm aware of run with a single etcd or with quorum reads on
Perfect, works for us. As for fixing this particular token controller bug, I no longer have skin in that game -- your call entirely. :-)


Question: // AttachDevice attaches the volume device to the host at a given mount path. func (u *storageosUtil) AttachDevice(b *storageosMounter, deviceMountPath string) error { <ins> klog.V(4).Infof("attaching StorageOS device for volume %q with namespace %q", b.volName, b.volNamespace) </ins> if err := u.NewAPI(b.apiCfg); err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a PVC is created and the PV mounted into a Pod using the StorageOS in-tree volume driver, the StorageOS CLI & UI do not report that the volume is mounted. The PV is created and mounted into the Pod successfully, and all k8s functionality works as expected. However, since the correct mount status is not visible to administrators using the StorageOS UI or CLI, they may make changes to or delete a mounted volume without realising, and the safeguards (must use to delete a mounted volume) do not trigger. This is a regression caused by , released in 1.14.0. What you expected to happen: The StorageOS CLI should report volume mount information (node name, mount timestamp). How to reproduce it (as minimally and precisely as possible): StorageOS node containers with CSI disabled: application Pod with a PVC using the StorageOS StorageClass: StorageOS to list volumes: Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 and later Cloud provider or hardware configuration: any OS (e.g: ): any linux (incl. Ubuntu 18.04) Kernel (e.g. ): any linux (incl. 4.15.0-33-generic) Install tools: any (incl. minikube) Network plugin and version (if this is a network-related bug): N/A Others: Requires StorageOS deployment /sig storage /assign


Question: // ResourcePolicyRule matches a resource request if and only if: (a) // at least one member of verbs matches the request, (b) at least one // member of apiGroups matches the request, (c) at least one member of <del> // resources matches the request, and (d) least one member of // namespaces matches the request. </del> <ins> // resources matches the request, and (d) either (d1) the request does // not specify a namespace (i.e., `Namespace==""`) and clusterScope is // true or (d2) the request specifies a namespace and least one member // of namespaces matches the request's namespace. </ins> message ResourcePolicyRule { // `verbs` is a list of matching verbs and may not be empty. // "*" matches all verbs and, if present, must be the only entry.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I looked at the comment on ResourcePolicyRule to understand exactly what its ClusterScope field means, and was dismayed to find an outdated precise description that does not mention that field. I expected that comment to be up to date. Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc /cc /cc /cc
/triage accepted


Question: name: "no scale down stabilization", key:  "", recommendations: []timestampedRecommendation{ <del> {1, time.Now().Add(-2 * time.Minute)}, {2, time.Now().Add(-1 * time.Minute)}}, </del> <ins> {1, now.Add(-2 * time.Minute)}, {2, now.Add(-1 * time.Minute)}}, </ins> currentReplicas:              100, // to apply scaleDown delay we should have current > desired prenormalizedDesiredReplicas: 3, expectedStabilizedReplicas:   3, expectedRecommendations: []timestampedRecommendation{ <del> {1, time.Now()}, {2, time.Now()}, {3, time.Now()}, </del> <ins> {1, now}, {2, now}, {3, now}, </ins> }, scaleUpStabilizationWindowSeconds: 60 * 5, },

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: }, nil } <ins> func (plugin *rbdPlugin) getDeviceNameFromOldMountPath(mounter mount.Interface, mountPath string) (string, error) { refs, err := mount.GetMountRefsByDev(mounter, mountPath) if err != nil { return "", err } // baseMountPath is the prefix of deprecated device global mounted path, // such as: /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd baseMountPath := filepath.Join(plugin.host.GetPluginDir(rbdPluginName), "rbd") for _, ref := range refs { if dstrings.HasPrefix(ref, baseMountPath) { return filepath.Rel(baseMountPath, ref) } } return "", fmt.Errorf("can't find source name from mounted path: %s", mountPath) } </ins> func (plugin *rbdPlugin) NewDeleter(spec *volume.Spec) (volume.Deleter, error) { if spec.PersistentVolume != nil && spec.PersistentVolume.Spec.RBD == nil { return nil, fmt.Errorf("spec.PersistentVolumeSource.Spec.RBD is nil")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The method for ceph rbd volume plugin generates the path with code as follows: So the device mount path's format will be: , this implementation has two problems: with other volume plugins, as all the other plugins return , see as an example. method of ceph rbd plugin is not working properly, because rbd plugin uses method to find the device mount path(just like all the other plugins do), and method assumes the device mount path's format is (see ). What you expected to happen: The device mount path of rbd plugin should be . Anything else we need to know?: Do we have any reason for this design? If not, I will submit a PR to fix this, the adjusted method maybe look like this: /sig storage
/assign
nice catch There are two ways to fix this: Change mount path. But this will introduce compatibility issues during upgrade. Use , instead of to replace PR welcome
cc
Hi , there is a difference between and : returns the device path, such as , and returns the device mounted path, such as . What we really want is the second one, as we need to find the pool and image name. So we can't just simply replace with . Maybe we can find the mounted path specially for rbd plugin, use , the code looks like this:
thanks let make a forward moving plan: fix and append to the global path keep in . If no path is found, fall back to old path scheme using your code
Nice idea , I've submit a PR for this, PTAL.
I've checked and the reason is that I tested with the old release branch, the has been changed with in mount path, and causes the problem. thanks for your PR
i read again. There is another return path . So in rbd, the missing dir will not make into , but the second return should still give us the dir we need
You are right , indeed has two return path, but the second return still can't meet our needs. Because the parameter of is the mount path for a specific pod, such as , its format is , so will return the volume name. This volume name is returned by (see ), usually is the PV's name or the name specified in , both of these are not .


Question: return r, err } if len(r.APIResources) == 0 { <del> return r, fmt.Errorf("Got empty response for: %v", groupVersion) </del> <ins> return r, &emptyResponseError{gv: groupVersion} </ins> } return r, nil }

I think this got stale and closed because it was posted on but is still very relevant: The created by memory.NewMemCacheClient(...) returns an error when calling ServerResourcesForGroupVersion if the APIResources list returned is empty, as seen here: Our particular case came up due to the API group being registered with the API server, but having no resources. If using like to discover with zero current resources there should not be an error returned. Do a discovery with a against a Kubernetes cluster that has at least one with zero resources. The condition on the server that triggers the described issue can be expressed as e.g.: I would be good to get to know the reasoning behind the emptiness check in the , maybe there is good reason for it. <detailsClient Version: v1.20.15 Server Version: v1.22.8-gke.202 </details<detailsGoogle Cloud (GKE) </detailsNo response No response No response No response
/sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
Reviving this issue based on This behavior has become quite annoying for the custom/external metrics aggregated APIs since after , client-go is now producing a very spammy error log when an aggregated API is found without resource. This is a very common scenario with the metric APIs since metrics are dynamically registered as Resources to the aggregated APIs so on startup the aggregated apiserver may not expose any resource and that's completely normal. It is even more common for implementations of these APIs that register metric Resources from CRDs as CRs tend to be created later than APIServices and the aggregated apiserver. I can see a few path forward to improve the situation for us: the error if we consider that having no resources exposed by an aggregated API is acceptable the verbosity of the log to just show it in debugging scenarios somehow update to defer registration until we have at least one Metric resource. could you perhaps share your insight on that one please? /remove-lifecycle stale
Hello folks! Any update on this topic? The message is really spammy
Removing triage accepted manually since there was no update on this for more than 6 months. /remove-triage accepted
For context, I traced it back and check has been considered an error since the first iteration of the in-memory discovery client in : It is unexpected for a built-in or CRD groupversion to have no resources, but for aggregated-apiservice I can see it being possible that the object is created and set up for discovery before the aggregated-apiservice is ready.
I think this is which introduces (but maybe I'm wrong). There is also in that repo about the same issue after upgrading the kube-client pkg. in custom-metrics repo be correct from Kubernetes pov? IMO we need to find a solution soon or the amount of noise will increase once people bump their deps and tooling (, kubectl is affected, and any tool which uses kube-client is affected)
/triage accepted


Question: if err := q.Delete(&highPriNominatedPod); err != nil { t.Errorf("delete failed: %v", err) } <ins> q.lock.RLock() </ins> if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&unschedulablePod)); !exists { t.Errorf("Expected %v to be in activeQ.", unschedulablePod.Name) } if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&highPriNominatedPod)); exists { t.Errorf("Didn't expect %v to be in activeQ.", highPriorityPod.Name) } <ins> q.lock.RUnlock() </ins> if len(q.nominatedPods.nominatedPods) != 1 { t.Errorf("Expected nomindatePods to have only 'unschedulablePod': %v", q.nominatedPods.nominatedPods) }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: pkg/scheduler/internal/queue Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /assign
I just sent a pr for it according to the error message, but I'm not very sure if it will make things all right.
/assign


Question: apiVersion: v1 kind: Pod metadata: <del> name: l7-lb-controller-v0.9.2 </del> <ins> name: l7-lb-controller-v0.9.3 </ins> namespace: kube-system labels: k8s-app: glbc <del> version: v0.9.2 </del> <ins> version: v0.9.3 </ins> kubernetes.io/cluster-service: "true" kubernetes.io/name: "GLBC" spec: terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/google_containers/glbc:0.9.2 </del> <ins> - image: gcr.io/google_containers/glbc:0.9.3 </ins> livenessProbe: httpGet: path: /healthz

This leads to the following errors when running the example Guest Book application: If the proxy and the endpoint are on the same machine then the proxy works because everything defaults to 127.0.0.1 Successful proxy logs
I have this problem too. I think this has something to do with not having pod_registry/fillPodInfo called on the pods, but I'm not sure how to get the code where it needs to be.
I have repro-d this, I should be able to close it down soon. Sorry! --brendan , Ryan Fowler wrote:
please see for the fix. Sorry for the issue!
This issue is resolved. Closing.


Question: // hostname or not, we only check for the suffix match. if strings.HasSuffix(image, tag) || strings.HasSuffix(tag, image) { return true <ins> } else { // TODO: We need to remove this hack when project atomic based // docker distro(s) like centos/fedora/rhel image fix problems on // their end. // Say the tag is "docker.io/busybox:latest" // and the image is "docker.io/library/busybox:latest" t, err := dockerref.ParseNormalizedNamed(tag) if err != nil { continue } // the parsed/normalized tag will look like // reference.taggedReference { // 	 namedRepository: reference.repository { // 	   domain: "docker.io", // 	   path: "library/busybox" //	}, // 	tag: "latest" // } // If it does not have tags then we bail out t2, ok := t.(dockerref.Tagged) if !ok { continue } // normalized tag would look like "docker.io/library/busybox:latest" // note the library get added in the string normalizedTag := t2.String() if normalizedTag == "" { continue } if strings.HasSuffix(image, normalizedTag) || strings.HasSuffix(normalizedTag, image) { return true } </ins> } } }

bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it.

So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it.

Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?