Question: SRCS=agnhost ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: } r.params = newParams url := r.URL() <ins> segments := strings.Split(r.URL().Path, "/") groupIndex := 0 index := 0 if r.URL() != nil && r.baseURL != nil && strings.Contains(r.URL().Path, r.baseURL.Path) { groupIndex += len(strings.Split(r.baseURL.Path, "/")) } if groupIndex >= len(segments) { return *url } const CoreGroupPrefix = "api" const NamedGroupPrefix = "apis" isCoreGroup := segments[groupIndex] == CoreGroupPrefix isNamedGroup := segments[groupIndex] == NamedGroupPrefix if isCoreGroup { // checking the case of core group with /api/v1/... format index = groupIndex + 2 } else if isNamedGroup { // checking the case of named group with /apis/apps/v1/... format index = groupIndex + 3 } else { // this should not happen that the only two possibilities are /api... and /apis..., just want to put an // outlet here in case more API groups are added in future if ever possible: // https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups // if a wrong API groups name is encountered, return the {prefix} for url.Path url.Path = "/{prefix}" url.RawQuery = "" return *url } //switch segLength := len(segments) - index; segLength { switch { // case len(segments) - index == 1: // resource (with no name) do nothing case len(segments)-index == 2: // /$RESOURCE/$NAME: replace $NAME with {name} segments[index+1] = "{name}" case len(segments)-index == 3: if segments[index+2] == "finalize" || segments[index+2] == "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+1] = "{name}" } else { // /namespace/$NAMESPACE/$RESOURCE: replace $NAMESPACE with {namespace} segments[index+1] = "{namespace}" } case len(segments)-index >= 4: segments[index+1] = "{namespace}" // /namespace/$NAMESPACE/$RESOURCE/$NAME: replace $NAMESPACE with {namespace},  $NAME with {name} if segments[index+3] != "finalize" && segments[index+3] != "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+3] = "{name}" } } url.Path = path.Join(segments...) </ins> return *url }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In Kubernetes 1.11.x, the controller-manager metrics are providing histogram records for every api endpoint ever called during namespace cleanup. For every namespace that is deleted since a controller has been the leader, 1000+ new lines of metrics are created / served indefinitely. When creating/deleting namespaces at a rate of 100/hour, that's 100,000+ new metrics lines being served for every request of controller-manager/metrics. spun up a 1.10 environment, and says it looks like the namespace names were replaced with {namespace}: In 1.11 this has changed (short clip of 1 histogram): What you expected to happen: Don't generate so many metrics. How to reproduce it (as minimally and precisely as possible): To replicate in 1.11.x, simply create and then delete a namespace and query the metrics endpoint of your controller-manager. Anything else we need to know?: Repro steps and bug reported by /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
did some more debugging and believes that the root cause is He says that PR switched to the new dynamic client so when a request is made, this function is called: He explains that function in the old client checks if is set, the following is an example call: In the new client, the following is an example call: In this new dynamic client, the delete operation calls: whereas in the old client, it ran: The , he says, is the critical component, which sets the request client namespace field without namespace being set, when is called then the metrics are populated without renaming the Namespace to which causes tons of unique metrics to be generated and cause metrics data grow very quickly. This seems to be a pretty critical bug. Will mark it for v1.12. Please triage. /milestone v1.12 CC
- friendly reminder that Code Freeze is next week; if this is critical enough for 1.12 it needs to be triaged accordingly.

This isn't a regression (1.11 had it), but it's a big enough bug that I'd take a fix for it in 1.12. I think that a fix should take the form of fixing the to handle cases where the abspath is set. That would fix it for all future callers instead of just a one off for this client. That method is only called for metrics and our api paths follow a very fixed format. I expect that it should be fairly easy to recognize and you only have to do the analysis if you have no hits for the namespace being set.
please apply a priority/critical-urgent label to keep track for 1.12
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I applied the label. We will have to un-apply it if a targeted fix doesn't show up very soon :)
If no one has started working on it, I am on this now. I will try fix it ASAP.
please do!
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
please update on a daily basis, as we're in Code Freeze. :)
I have reproduced the problem and also wrote a unit test case for confirmation. Basically with the current dynamic client implementation, the actually string of namespace is in url here: , so when finalURLTemplate function checks (), it returns false and would not replace namespace with "{namespace}". I am working on fixing it by replacing the url segment after /namespaces/ by . I should be able to submit the PR tomorrow if not today.


Question: ``` The first example selects all resources with key equal to `environment` and value equal to `production` or `qa`. <del> The second example selects all resources with key equal to `tier` and value other than `frontend` and `backend`. </del> <ins> The second example selects all resources with key equal to `tier` and values other than `frontend` and `backend`. </ins> The third example selects all resources including a label with key `partition`; no values are checked. <del> Similarly the comma separator acts as an _AND_ operator for example filtering resource with a `partition` key (not matter the value) and with `environment` different than  `qa`. For example: `partition,environment notin (qa)`. </del> <ins> Similarly the comma separator acts as an _AND_ operator. So filtering resources with a `partition` key (no matter the value) and with `environment` different than  `qa` can be achieved using `partition,environment notin (qa)`. </ins> The _set-based_ label selector is a general form of equality since `environment=production` is equivalent to `environment in (production)`; similarly for `!=` and `notin`. _Set-based_ requirements can be mixed with _equality-based_ requirements. For example: `partition in (customerA, customerB),environment!=qa`.

The suggests we support both "equality-based" and "set-based" label selectors, but actually we only support "==" selector right now. This is confusing. People are asking questions on IRC. We should update the doc in both github and
cc
We support the set-based selectors for filtering GET/watch requests. We support equality-based selectors, represented as maps, for API resources, such as ReplicationController, Service, and Pod nodeSelector.
Even if it's technically correct people are getting confused, so we need to fix the documentation.
Ref
sorry I see this only now. I'm ok to better explain the difference between set-based and equality based but I would need of some example of questions on IRC. May you have some of them? Otherwise we may clearly write that kubectl <command-l <selectorsupports set-based and equality based selector while API resources supports only equality-based.
I was thinking of making the difference between selectors in kubectl and in API clear.
cool, I can work on this if you've not already started.
By all means - please do:)
:)


Question: return "", nil } <del> type dockerPuller struct{} </del> <ins> type dockerPuller struct { endpoint string } </ins> <del> func MakeDockerPuller() DockerPuller { return dockerPuller{} </del> <ins> func MakeDockerPuller(endpoint string) DockerPuller { return dockerPuller{ endpoint: endpoint, } </ins> } <del> func (dockerPuller) Pull(image string) error { cmd := exec.Command("docker", "pull", image) </del> <ins> func (p dockerPuller) Pull(image string) error { var cmd *exec.Cmd if len(p.endpoint) == 0 { cmd = exec.Command("docker", "pull", image) } else { cmd = exec.Command("docker", "-H", p.endpoint, "pull", image) } </ins> err := cmd.Start() if err != nil { return err

When using or with a key/value that has a multi-line string as value, that value will appear as quoted string in the log message, i.e. the log message will still be a single line. It would be nicer to print the value in text mode so that it goes across multiple lines, with clear indicators for start and end of the value. Indention could be used to avoid mistaking some embedded line as the start of a new log message. In JSON mode, the value should be formatted normally. /sig instrumentation /wg structured-logging Always printing as a single line can can make the log output unreadable ().
When adding something like this, can be reverted and implemented differently.
My original thinking was that only objects that explicitly ask for multi-line output get formatted that way. But that means that we need to review log calls very carefully. An alternative is to automatically switch from quoting strings to multi-line output based on some heuristic, for example when there's more than one line break.
Here's a potential implementation:
If structured logging is needed in any other case I can contribute, please inform if needed.
/triage accepted /assign /priority important-soon /area logging /cc /cc
We have the same problem as in kube-scheduler also in kubelet, it's just that no-one has seen and/or complained about the multi-line strings. With the benchmark + log analysis tool from I found the following multi-line string parameters in a kubelet log: <details</details


Question: } // nextPodOrDie returns the next Pod in the scheduler queue. <del> // The operation needs to be completed within 15 seconds; otherwise the test gets aborted. </del> <ins> // The operation needs to be completed within 5 seconds; otherwise the test gets aborted. </ins> func nextPodOrDie(t *testing.T, testCtx *testutils.TestContext) *framework.QueuedPodInfo { t.Helper() var podInfo *framework.QueuedPodInfo // NextPod() is a blocking operation. Wrap it in timeout() to avoid relying on // default go testing timeout (10m) to abort. <del> if err := timeout(testCtx.Ctx, time.Second*15, func() { </del> <ins> if err := timeout(testCtx.Ctx, time.Second*5, func() { </ins> podInfo = testCtx.Scheduler.NextPod() }); err != nil { t.Fatalf("Timed out waiting for the Pod to be popped: %v", err) } return podInfo } <ins> // nextPod returns the next Pod in the scheduler queue, with a 5 seconds timeout. func nextPod(t *testing.T, testCtx *testutils.TestContext) *framework.QueuedPodInfo { t.Helper() var podInfo *framework.QueuedPodInfo // NextPod() is a blocking operation. Wrap it in timeout() to avoid relying on // default go testing timeout (10m) to abort. if err := timeout(testCtx.Ctx, time.Second*5, func() { podInfo = testCtx.Scheduler.NextPod() }); err != nil { return nil } return podInfo } </ins>

Currently, uses ServiceAffinity plugin to cover the event registration logic: But along with the removal of ServiceAffinity plugin, this test will also be removed . We will come up with a supplemental test to cover the event registration logic. /sig scheduling
/assign /triage accepted
/kind cleanup


Question: import ( "fmt" <ins> "reflect" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/conversion" </ins> ) func addConversionFuncs() { <del> err := api.Scheme.AddConversionFuncs() </del> <ins> err := api.Scheme.AddConversionFuncs( convert_api_StatusDetails_To_v1_StatusDetails, convert_v1_StatusDetails_To_api_StatusDetails, ) </ins> if err != nil { // If one of the conversion functions is malformed, detect it immediately. panic(err)

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: currentLine++ } if currentLine == len(outputLines) { <del> framework.Failf("Failed to find %s in %s", requirement[0], output) </del> <ins> return fmt.Errorf("failed to find %s in %s", requirement[0], output) </ins> } for _, item := range requirement[1:] { if !strings.Contains(outputLines[currentLine], item) { <del> framework.Failf("Failed to find %s in %s", item, outputLines[currentLine]) </del> <ins> return fmt.Errorf("failed to find %s in %s", item, outputLines[currentLine]) </ins> } } } <ins> return nil } func checkOutput(output string, required [][]string) { err := checkOutputReturnError(output, required) if err != nil { framework.Failf("%v", err) } } func checkKubectlOutputWithRetry(required [][]string, args ...string) { var pollErr error wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie(args...) err := checkOutputReturnError(output, required) if err != nil { pollErr = err return false, nil } pollErr = nil return true, nil }) if pollErr != nil { framework.Failf("%v", pollErr) } return </ins> } func getAPIVersions(apiEndpoint string) (*metav1.APIVersions, error) {

https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing.

I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close


Question: return errs } <del> func newEffectiveVersion(binaryVersion *version.Version) MutableEffectiveVersion { </del> <ins> func newEffectiveVersion(binaryVersion *version.Version, useDefaultBuildBinaryVersion bool) MutableEffectiveVersion { </ins> effective := &effectiveVersion{} compatVersion := binaryVersion.SubtractMinor(1) effective.Set(binaryVersion, binaryVersion, compatVersion) <ins> effective.useDefaultBuildBinaryVersion.Store(useDefaultBuildBinaryVersion) </ins> return effective }

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: // scheduler framework will skip Filter if PreFilter fails return } <del> t.Logf("check state after prefilter phase") </del> <ins> t.Logf("Verify: check state after prefilter phase") </ins> stateData, err := getStateData(state) if err != nil { t.Fatal(err)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: The unit test Which test(s) are flaking: Testgrid link: Reason for failure: In volume scheduling, we maintain internal caches that watch for PV/PVC events. However, we don't have a method to wait for them to be synced in UT. When the cache in the informer factory is synced but the execution of registered handlers may not finish. <!-- In production, this is not a problem. Even if the scheduler marks the pod uncacheable due to the stale cache in PVAssumeCache/PVCAssumeCache, the pod will be moved to the active queue later as the scheduler . related: --Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig scheduling /sig storage
Even if I check the PV/PVC assume cache and wait for PV/PVC objects to be into the cache, the PV/PVC objects may not be . It seems that the shared informer with the fake client has a bug that registered event handlers may not be called (or handlers are not registered successfully).
The share informer with the fake client may miss events of objects which are created before caches are synced. it's better to create testing objects before starting the informers
Although the event handlers may not finish when the caches were synced in theory, however, it seems it's not necessary to wait for PV/PVC assume cache to be synced practically. I've run a thousand times (parallel: 10) with locally and no failures.


Question: t.Errorf("Expected binder %q but got %q", expectedBinderType, binderType) } } <ins> func TestNodeAllocatableChanged(t *testing.T) { newQuantity := func(value int64) resource.Quantity { return *resource.NewQuantity(value, resource.BinarySI) } for _, c := range []struct { Changed        bool OldAllocatable v1.ResourceList NewAllocatable v1.ResourceList }{ // No allocatable resources changed. { Changed:        false, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, }, // New node has more allocatable resources. { Changed:        true, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024), v1.ResourceStorage: newQuantity(1024)}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.OldAllocatable}} newNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.NewAllocatable}} changed := nodeAllocatableChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeAllocatableChanged should be %t, got %t", c.Changed, changed) } } } func TestNodeLabelsChanged(t *testing.T) { for _, c := range []struct { Changed   bool OldLabels map[string]string NewLabels map[string]string }{ // No labels changed. {Changed: false, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar"}}, // Labels changed. {Changed: true, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar", "test": "value"}}, } { oldNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.OldLabels}} newNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.NewLabels}} changed := nodeLabelsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeLabelsChanged should be %t, got %t", c.Changed, changed) } } } func TestNodeTaintsChanged(t *testing.T) { for _, c := range []struct { Changed        bool OldTaints      []v1.Taint NewTaints      []v1.Taint OldAnnotations map[string]string NewAnnotations map[string]string }{ // Taints use annotation and no change. { Changed:        false, OldAnnotations: map[string]string{core.TaintsAnnotationKey: `[{"key":"value"}]`}, NewAnnotations: map[string]string{core.TaintsAnnotationKey: `[{"key":"value"}]`}, }, // Taints use annotation and changed. { Changed:        true, OldAnnotations: map[string]string{core.TaintsAnnotationKey: `[{"key":"value1"}]`}, NewAnnotations: map[string]string{core.TaintsAnnotationKey: `[{"key":"value2"}]`}, }, // Taints use Spec.Taints and no change. { Changed:   false, OldTaints: []v1.Taint{{Key: "key", Value: "value"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value"}}, }, // Taints use Spec.Taints and changed. { Changed:   true, OldTaints: []v1.Taint{{Key: "key", Value: "value1"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value2"}}, }, } { oldNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Annotations: c.OldAnnotations}, Spec: v1.NodeSpec{Taints: c.OldTaints}} newNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Annotations: c.NewAnnotations}, Spec: v1.NodeSpec{Taints: c.NewTaints}} changed := nodeTaintsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeTaintsChanged should be %t, not %t", c.Changed, changed) } } } func TestNodeConditionsChanged(t *testing.T) { for _, c := range []struct { Changed       bool OldConditions []v1.NodeCondition NewConditions []v1.NodeCondition }{ // No conditions changed. { Changed:       false, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, }, // New node has more healthy conditions. { Changed:       true, OldConditions: []v1.NodeCondition{}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, // NodeReady False -> True { Changed:       true, OldConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.OldConditions}} newNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.NewConditions}} changed := nodeConditionsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeConditionsChanged should be %t, got %t", c.Changed, changed) } } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) <ins> testKubelet.fakeCadvisor.On("VersionInfo").Return(&cadvisorapi.VersionInfo{}, nil) testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorapi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("ImagesFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) </ins> kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new") containers := []api.Container{{Name: "succeed"}, {Name: "failed"}}

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: const bindPluginAnnotation = "bindPluginName" func (bp *BindPlugin) Name() string { <del> return bp.PluginName </del> <ins> return bp.name </ins> } func (bp *BindPlugin) Bind(ctx context.Context, state *framework.CycleState, p *v1.Pod, nodeName string) *framework.Status {

See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: <del> #!/usr/bin/env bash # Copyright 2022 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This scripts invokes `kind build image` so that the resulting # image has a containerd with CDI support. # # Usage: kind-build-image.sh <tag of generated image> set -ex set -o pipefail # TODO (betheelder,pohly): this script is a bit of a no-op now, but people may # still be referencing it (CI is not). Clean up later. tag="$1" kind build node-image --image "$tag" "$(pwd)" </del>

uses containerd for worker nodes by default, and latest release of uses by default base-image with containerd 1.6 for worker node. Unfortunately, CDI was merged into containerd 1.7, and CI at the moment uses binary from unreleased build 0.20-alpha. So running e2e tests locally for is failing unless when building node image a parameter is used with base-image that has 1.7 containerd. The need needs to be mentioned in DRA e2e Dynamic Resource Allocation e2e work locally when running them according to the instruction. Run DRA e2e with kind 0.19. The fix is coming shortly. 27, 1.28-alpha N/A N/A N/A N/A N/A
/sig node /assign
/triage accepted
I recommend testing kubernetes @ HEAD with KIND @ HEAD more generally. Nearly all of Kubernetes CI does this, we can't guarantee that a previously released binary has fixes to handle unreleased kubernetes breaking changes so periodically this is required regardless of this specific issue.
Alright, that recommendation was not noticed by me unfortunately. And thinking that there might be more like me, let's at least document it in the DRA e2e tests README.


Question: github.com/kr/pty => github.com/kr/pty v1.1.5 github.com/kr/text => github.com/kr/text v0.1.0 github.com/kylelemons/godebug => github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 <del> github.com/libopenstorage/openstorage => github.com/libopenstorage/openstorage v0.0.0-20170906232338-093a0c388875 </del> <ins> github.com/libopenstorage/openstorage => github.com/libopenstorage/openstorage v1.0.0 </ins> github.com/liggitt/tabwriter => github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de github.com/lithammer/dedent => github.com/lithammer/dedent v1.1.0 github.com/lpabon/godbc => github.com/lpabon/godbc v0.1.1

Found this when trying to see which vendored dependencies does not use a proper tag (and is using a random SHA). The dependency is very old and there have been a lot of updates to it: Looks like the However there does not seem to be anyone around to take care of the portworx plugin. Is it time to deprecate it? is anyone using it? /sig storage
cc ?
/area code-organization
Hello The portworx volume plugin is definitely in use. The portworx volume plugin doesn't need frequent updates since the plugin acts as a simple client to all provisioning calls. In the same view, the vendored openstorage package doesn't need to be updated too. The server side implementation in Portworx handles all the latest Portworx features.
cc:
in that case, can you please update the dependency to a specific tag instead of a random SHA?
/assign updated the title of the issue as well
my request is not for a new release or tag in libopenstorage/openstorage repository. Just to update the SHA we use in our k/k repository to one of the released versions ( ). This is not asking or demanding anything to be done in libopenstorage/openstorage, just that we should update what we use from there (and avoid a random SHA). We are trying to do this for all vendored libraries (one by one … slowly as part of the sig-arch code organization sub project, so it’s not a random request)
/assign
GitHub didn't allow me to assign the following users: lpabon. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After a chat with - apparently this request is coming across the wrong way I am closing this out as it seems to add a over head /close
Closing this issue. <detailsIn response to : After a chat with - apparently this request is coming across the wrong way /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "managedFields": [{ "manager": "apply", "operation": "Apply", <del> "apiVersion": "v1", </del> <ins> "apiVersion": "apps/v1", </ins> "fields": { "f:metadata": { "f:labels": {

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: ) fp.syncProxyRules() <del> if fexec.CommandCalls != 2 { t.Fatalf("Updated UDP service with new endpoints must clear UDP entries") </del> <ins> if fexec.CommandCalls != 4 { t.Fatalf("Updated UDP service with new endpoints must clear UDP entries 4 times: ClusterIP, NodePort, ExternalIP and LB") </ins> } <del> // Delete ClusterIP Conntrack entries expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s -p %s", svcIP, strings.ToLower(string((v1.ProtocolUDP)))) actualCommand := strings.Join(fcmd.CombinedOutputLog[0], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) </del> <ins> // the order is not guaranteed so we have to compare the strings in any order expectedCommands := []string{ // Delete ClusterIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", svcIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete ExternalIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", extIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete LoadBalancerIP Conntrack entries fmt.Sprintf("conntrack -D --orig-dst %s -p %s", lbIngressIP, strings.ToLower(string((v1.ProtocolUDP)))), // Delete NodePort Conntrack entrie fmt.Sprintf("conntrack -D -p %s --dport %d", strings.ToLower(string((v1.ProtocolUDP))), nodePort), </ins> } <del> // Delete NodePort Conntrack entrie expectCommand = fmt.Sprintf("conntrack -D -p %s --dport %d", strings.ToLower(string((v1.ProtocolUDP))), nodePort) actualCommand = strings.Join(fcmd.CombinedOutputLog[1], " ") if actualCommand != expectCommand { t.Errorf("Expected command: %s, but executed %s", expectCommand, actualCommand) </del> <ins> actualCommands := []string{ strings.Join(fcmd.CombinedOutputLog[0], " "), strings.Join(fcmd.CombinedOutputLog[1], " "), strings.Join(fcmd.CombinedOutputLog[2], " "), strings.Join(fcmd.CombinedOutputLog[3], " "), } sort.Strings(expectedCommands) sort.Strings(actualCommands) if !reflect.DeepEqual(expectedCommands, actualCommands) { t.Errorf("Expected commands: %v, but executed %v", expectedCommands, actualCommands) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --UDP traffic to a Pod behind a LoadBalancer Service exposing a UDP listener service is blackholed when the replacement Pod is scheduled on the same node as the "old" Pod. A new Pod that gets provisioned (replacing the previous one) will receive traffic once it's ready. a UDP Listener Pod (single pod) and using nodeSelector ensure it runs on a specific node the Pod behind a LoadBalancer Service continuous stream of UDP packets from outside the cluster to the External IP of the Service that traffic is correctly received the Pod and observe as a new one is created, traffic should now be blackholed and not reaching the "new" Pod After the new Pod is recreated, we noticed the following entries on the node: notice that the <SVC EXT IPappears here both in the original and the NATed , this is wrong and I suspect happens because the conntrack for the old Pod were flushed before the iptables rules for the new Pod were created (?) and in the interim new packets have reached the node and resulted in these incorrect conntrack entries. To "fix" this, we manually deleted the above conntrack entries, and this allowed new correct one to be established based on the now existing iptables nat rule: with this new conntrack entries, the traffic started flowing as expected to the new Pod and notice that the NATed now correctly shows the Pod IP (172.30.3.219) instead of the Service External IP. While spelunking through the GitHub issues and PRs, I found this issue which was addressed in , it does mention that: the service has endpoints available. Since the fix in this PR refers to NodePort, perhaps the same also applies to External IP (i.e. LoadBalancer Service?) or there's something else at play here? AFAICT, this fix is included in v1.20.7 that we are running. Kubernetes version (use ):Cloud provider or hardware configuration: GCP/GKE OS (e.g: ):Kernel (e.g.

): Install tools: Terraform (with google provider) Network plugin and version (if this is a network-related bug): Others:
/sig network
According to GCP support (since we hit this on a GKE cluster) there's already a fix for this bug and it in v1.21, however I'm unable to find to link it here (hope they can provide the link shortly).
GCP support referenced as the fix for this issue, however this was already merged and released in v1.20.4 but we are running v1.20.7, so should have the fix already, so the status is unclear at this point.
thanks for the detailed report, at first sight we do clean the external ips is it possible to set kube-proxy in a log level of 2 and capture the logs from the node to correlate with the time it happends, and understand why those entries are not being deleted?
Hey we can try that, however since this is a GKE environment I'd have to check how/if we can change kube-proxy log level. My understanding of the issue is that it's not that the conntrack entries are flushed, is that they are flushed before the nat iptables rules are created, causing faulty conntrack entries to be created in the interim (since the udp packets are incoming all the time).
I'm having some miscommunication about the availability of fix in with GCP/GKE support and was hoping you could confirm that is included in v1.20.5 (as per the release notes there? ) and per the cherry-pick PR GCP Support claim that this change is incorporated in v1.21 and we should wait for v1.21 to be available in GKE.
I see the commit in 1.20.5 , this is the cherry-pick The conntrack entries are always deleted after the conntrack entries are created, there is some code that detects that a service goes stale and based on that it deletes the entries.

I think that current code detects this case, at least I know it works when the service goes from 0 to 1 endpoint, that is why the logs will be useful, we can see exactly what happens ...
Thanks for confirming I just checked and seem to be already running in log level 2: I looked back at the logs when this issue was identified and I see the following entries: This two messages and only these two repeat every time we tried to rollout the deployment. is indeed the and is the ClusterIP of the service: but every time we tried, we ended up with these stale conntrack entries I mentioned in the OP and only after deleting them would the udp packet start flowing into the "new" pod.
oh wait, by external IP you mean ?
/triage accepted /assign
here you go, having good reports on bugs make them easy to spot ;)
indeed! I apologize if I wasn't clear indeed I was referring to the LoadBalancer Service External IP. Thanks for fix!!! But now I'm a bit confused, since I understood that LoadBalancer Service External IP == LoadBalancer Ingress IP, but I see these are two separate attributes? so for example: These are the same IP, but internally they are separate attributes?
hmm I don't know how kubectl prints the information, but you can check the fields with and you could see that ExternalIPs and LoadBalancer.IngressIP are different fields, actually one is in the Status and other in the Spec
interesting, so in the output, I can only see the , there's actually no mention of here: That's why I always assumed that this is referring to the same thing/attribute/field.


Question: return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(sandboxID string) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(sandboxID string, sandbox *dockertypes.ContainerJSON) string { // Versions and feature support // ============================ // Windows version >= Windows Server, Version 1709, Supports both sandbox and non-sandbox case // Windows version == Windows Server 2016 Support only non-sandbox case // Windows version < Windows Server 2016 is Not Supported // Sandbox support in Windows mandates CNI Plugin. // Presence of CONTAINER_NETWORK flag is considered as non-Sandbox cases here // Hyper-V isolated containers are also considered as non-Sandbox cases // Todo: Add a kernel version check for more validation // Hyper-V only supports one container per Pod yet and the container will have a different // IP address from sandbox. Retrieve the IP from the containers as this is a non-Sandbox case. // TODO(feiskyer): remove this workaround after Hyper-V supports multiple containers per Pod. if networkMode := os.Getenv("CONTAINER_NETWORK"); networkMode == "" && sandbox.HostConfig.Isolation != kubeletapis.HypervIsolationValue { // Sandbox case, fetch the IP from the sandbox container. return ds.getIP(sandboxID, sandbox) } // Non-Sandbox case, fetch the IP from the containers within the Pod. </ins> opts := dockertypes.ContainerListOptions{ All:     true, Filters: dockerfilters.NewArgs(),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?

the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call.

So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?


Question: ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, <ins> "Verify we create a pod sandbox if no ready sandbox for pod with RestartPolicy=Never and no containers have ever been created": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { // no ready sandbox status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.SandboxStatuses[0].Metadata.Attempt = uint32(2) // no visible containers status.ContainerStatuses = []*kubecontainer.ContainerStatus{} }, actions: podActions{ SandboxID:         baseStatus.SandboxStatuses[0].Id, Attempt:           uint32(3), CreateSandbox:     true, KillPod:           true, ContainersToStart: []int{0, 1, 2}, ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, </ins> "Kill and recreate the container if the container is in unknown state": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever. What you expected to happen: Since the user/application containers were never created, kubelet should retry the PodSandbox creation. The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user. How to reproduce it (as minimally and precisely as possible): I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations. Anything else we need to know?: Relevant code snippet: Environment: Kubernetes version (use ): Only checked the master branch (1.16), but all supported release branches should have the issue Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues. Others:
thanks for posting! I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. I was thinking something similar to: Thoughts? Is there a simplex fix?
Hi! Thank you for filing this - it matches the behaviour we've been seeing. Am I correct that this is a regression from (), and so only affects Kubernetes =1.13 ?
Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though.
Thanks. That's probably correct. I saw this in happening in 1.13
Gotcha - thanks! Taking a look at your pr now :)


Question: compiler := authenticationcel.NewCompiler(environment.MustBaseEnvSet(environment.DefaultCompatibilityVersion(), true)) state := &validationState{} <del> allErrs = append(allErrs, validateIssuer(authenticator.Issuer, disallowedIssuers, fldPath.Child("issuer"))...) </del> <ins> allErrs = append(allErrs, validateIssuer(authenticator.Issuer, disallowedIssuers, fldPath.Child("issuer"), structuredAuthnFeatureEnabled)...) </ins> allErrs = append(allErrs, validateClaimValidationRules(compiler, state, authenticator.ClaimValidationRules, fldPath.Child("claimValidationRules"), structuredAuthnFeatureEnabled)...) allErrs = append(allErrs, validateClaimMappings(compiler, state, authenticator.ClaimMappings, fldPath.Child("claimMappings"), structuredAuthnFeatureEnabled)...) allErrs = append(allErrs, validateUserValidationRules(compiler, state, authenticator.UserValidationRules, fldPath.Child("userValidationRules"), structuredAuthnFeatureEnabled)...)

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question: <ins> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "fmt" "testing" "time" federation_api "k8s.io/kubernetes/federation/apis/federation/v1beta1" federation_release_1_4 "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_4" fake_federation_release_1_4 "k8s.io/kubernetes/federation/client/clientset_generated/federation_release_1_4/fake" api_v1 "k8s.io/kubernetes/pkg/api/v1" pkg_runtime "k8s.io/kubernetes/pkg/runtime" "github.com/stretchr/testify/assert" ) // Fake federation view. type fakeFederationView struct { } func (f fakeFederationView) GetClientsetForCluster(clusterName string) (federation_release_1_4.Interface, error) { return &fake_federation_release_1_4.Clientset{}, nil } func (f *fakeFederationView) GetReadyClusters() ([]*federation_api.Cluster, error) { return []*federation_api.Cluster{}, nil } func (f *fakeFederationView) GetReadyCluster(name string) (*federation_api.Cluster, bool, error) { return nil, false, nil } func (f *fakeFederationView) ClustersSynced() bool { return true } func TestFederatedUpdaterOK(t *testing.T) { addChan := make(chan string, 5) updateChan := make(chan string, 5) updater := NewFederatedUpdater(&fakeFederationView{}, func(_ federation_release_1_4.Interface, obj pkg_runtime.Object) error { clusterName, _ := GetClusterName(obj) addChan <- clusterName return nil }, func(_ federation_release_1_4.Interface, obj pkg_runtime.Object) error { clusterName, _ := GetClusterName(obj) updateChan <- clusterName return nil }, noop) err := updater.Update([]FederatedOperation{ { Type: OperationTypeAdd, Obj:  makeService("A", "s1"), }, { Type: OperationTypeUpdate, Obj:  makeService("B", "s1"), }, }, time.Minute) assert.NoError(t, err) add := <-addChan update := <-updateChan assert.Equal(t, "A", add) assert.Equal(t, "B", update) } func TestFederatedUpdaterError(t *testing.T) { updater := NewFederatedUpdater(&fakeFederationView{}, func(_ federation_release_1_4.Interface, obj pkg_runtime.Object) error { return fmt.Errorf("boom") }, noop, noop) err := updater.Update([]FederatedOperation{ { Type: OperationTypeAdd, Obj:  makeService("A", "s1"), }, { Type: OperationTypeUpdate, Obj:  makeService("B", "s1"), }, }, time.Minute) assert.Error(t, err) } func TestFederatedUpdaterTimeout(t *testing.T) { start := time.Now() updater := NewFederatedUpdater(&fakeFederationView{}, func(_ federation_release_1_4.Interface, obj pkg_runtime.Object) error { time.Sleep(time.Minute) return nil }, noop, noop) err := updater.Update([]FederatedOperation{ { Type: OperationTypeAdd, Obj:  makeService("A", "s1"), }, { Type: OperationTypeUpdate, Obj:  makeService("B", "s1"), }, }, time.Second) end := time.Now() assert.Error(t, err) assert.True(t, start.Add(10*time.Second).After(end)) } func makeService(cluster, name string) *api_v1.Service { return &api_v1.Service{ ObjectMeta: api_v1.ObjectMeta{ Namespace: "ns1", Name:      name, Annotations: map[string]string{ ClusterNameAnnotation: cluster, }, }, } } func noop(_ federation_release_1_4.Interface, _ pkg_runtime.Object) error { return nil } </ins>

Federated controllers will oftentimes have to crreate/update/delete the same/similar structures in multiple clusters. Given that the roundtrip time to remote clusters can be high (like, lets say, 1 sec) these operations should be done in parallel so that the main controller goroutine is not blocked for long. Part of


Question: e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")' initial_etcd_cluster: '$(echo "${INITIAL_ETCD_CLUSTER:-}" | sed -e "s/'/''/g")' <del> etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ:-}" | sed -e "s/'/''/g")' </del> hostname: $(hostname -s) EOF

Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Question: } // Run once for health check traffic <del> err = c.updateInstanceSecurityGroupsForNLBTraffic(actualGroups, desiredGroupIds, healthCheckPorts, lbName, []string{aws.StringValue(vpcCidr)}, false) </del> <ins> err = c.updateInstanceSecurityGroupsForNLBTraffic(actualGroups, desiredGroupIds, healthCheckPorts, lbName, vpcCidrBlocks, false) </ins> if err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When an AWS NLB is created through the Service of type, the health check security group rule is only created with a VPC primary CIDR block, as seen here: If the VPC has multiple CIDR blocks associated, and the worker nodes are running on those corresponding subnets, the health check fails. What you expected to happen: Multiple health check security group rules are created, one for each VPC CIDR block, using both and (). How to reproduce it (as minimally and precisely as possible): a VPC with with additional subnets created on a non-primary CIDR. a Kubernetes cluster (e.g. EKS) on subnets associated with a non-primary CIDR. a Service of type with nodes are healthy due to the security group rule being created only allowing traffic from the primary CIDR. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Amazon Linux 2 Kernel (e.g. ): Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, i'll provide a fix next week


Question: // NewAllocatorCIDRRange creates a Range over a net.IPNet, calling allocatorFactory to construct the backing store. func NewAllocatorCIDRRange(cidr *net.IPNet, allocatorFactory allocator.AllocatorFactory) (*Range, error) { <ins> registerMetrics() </ins> max := utilnet.RangeSize(cidr) base := utilnet.BigForIP(cidr.IP) rangeSpec := cidr.String()

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --Add metrics to apiserver tracking the service IP allocator - around how many IP addresses are used(allocated) vs available, and/or if there were any errors allocating service IPs to a service. We want to track if we are approaching the limit of the ServiceClusterIPRange, which is a fixed CIDR and difficult to change. Currently while we can approximate the usage looking at services in the cluster, it is not coming from the source allocator which is within the apiserver. Similar metrics are currently available for the node's ipam allocatior: /area apiserver /sig instrumentation /sig network
/triage accepted /assign I see something like this, gauges for and can you expand on this? a counter incremented each time an IP allocation fails?
that is one option. for contrast the nodeipam metric has an 'allocationtriesper_request' metric - i was imagining something similar to an 'attempts', but with a success:true/false tag/facet so we could compare how often allocations are attempted/how often they fail. I will defer to the experts here as I'm not sure what the status quo for reporting attempts/failures are. I'm not sure if there's any code paths that would cause allocation failures other than usage total (or available <= 0) - but having them separated out like that would be handy in that scenario. (maybe etcd internal allocation is corrupt? or context deadline exceeded)


Question: tags = ["automanaged"], deps = [ "//pkg/api:go_default_library", <ins> "//pkg/api/v1:go_default_library", </ins> "//pkg/apis/componentconfig:go_default_library", "//pkg/apis/componentconfig/v1alpha1:go_default_library", "//pkg/kubelet/qos:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.


Question: // Its corresponding flag only gets registered in Windows builds. WindowsService bool <del> // containerized should be set to true if kubelet is running in a container. Containerized bool </del> // remoteRuntimeEndpoint is the endpoint of remote runtime service RemoteRuntimeEndpoint string // remoteImageEndpoint is the endpoint of remote image service

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } } if err != nil { <del> klog.Errorf("azureDisk - detach disk(%s, %s)) failed, err: %v", diskName, diskURI, err) </del> <ins> klog.Errorf("azureDisk - detach disk(%s, %s) failed, err: %v", diskName, diskURI, err) </ins> } else { klog.V(2).Infof("azureDisk - detach disk(%s, %s) succeeded", diskName, diskURI) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add more logging in azure disk attach/detach What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
The issue message here doesn't say why a change was desired. The PR "Original logging is not clear when dealing with managed disk, since diskName is empty". That's more informative.


Question: GCE_HOSTS= GCE_IMAGE_CONFIG_PATH=test/e2e_node/jenkins/image-config.yaml GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins-pull </del> <ins> GCE_PROJECT=k8s-jkns-pr-node-e2e </ins> CLEANUP=true GINKGO_FLAGS='--skip="[Flaky]|[Slow]|[Serial]"' SETUP_NODE=false

I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.


Question: "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" <ins> "encoding/base64" </ins> "github.com/onsi/ginkgo" <ins> "k8s.io/apimachinery/pkg/types" </ins> ) var _ = ginkgo.Describe("[sig-api-machinery] Secrets", func() {

This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the follow endpoints related to Secrets. We chose the follow endpoints for this mock test. Test outline: a Secret in the default namespace with custom values Secrets in all namespaces search for Secret created in  the data in Secret  Secret  Secrets in all namespaces ensure that Secret  doesn't exist Discover useragents Display endpoints hit by the test Review the change in coverage If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close
Closing this issue. <detailsIn response to : Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if [[ -n "${NON_MASQUERADE_CIDR:-}" ]]; then flags+=" --non-masquerade-cidr=${NON_MASQUERADE_CIDR}" fi <ins> # FlexVolume plugin if [[ -n "${VOLUME_PLUGIN_DIR:-}" ]]; then flags+=" --volume-plugin-dir=${VOLUME_PLUGIN_DIR}" fi </ins> if [[ "${ENABLE_MANIFEST_URL:-}" == "true" ]]; then flags+=" --manifest-url=${MANIFEST_URL}" flags+=" --manifest-url-header=${MANIFEST_URL_HEADER}"

For FlexVolume, there is a kubelet flag, --volume-plugin-dir, that specifies the path which FlexVolume plugin uses to find the custom volume driver. Currently it's not possible to set this flag through cluster/kube-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage


Question: // ApplyTo applies the run options to the method receiver and returns self func (s *ServerRunOptions) ApplyTo(c *server.Config) error { c.CorsAllowedOriginList = s.CorsAllowedOriginList <ins> c.HSTSDirectives = s.HSTSDirectives </ins> c.ExternalAddress = s.ExternalHost c.MaxRequestsInFlight = s.MaxRequestsInFlight c.MaxMutatingRequestsInFlight = s.MaxMutatingRequestsInFlight

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: import ( "os" <del> "k8s.io/kubernetes/pkg/api" </del> dockertypes "github.com/docker/engine-api/types" <ins> dockercontainer "github.com/docker/engine-api/types/container" "k8s.io/kubernetes/pkg/api" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" </ins> ) <ins> // These two functions are OS specific (for now at least) func updateHostConfig(hc *dockercontainer.HostConfig, opts *kubecontainer.RunContainerOptions) { // There is no /etc/resolv.conf in Windows, DNS and DNSSearch options would have to be passed to Docker runtime instead hc.DNS = opts.DNS hc.DNSSearch = opts.DNSSearch // MemorySwap == -1 is not currently supported in Docker 1.14 on Windows // https://github.com/docker/docker/blob/master/daemon/daemon_windows.go#L175 hc.Resources.MemorySwap = 0 } func DefaultMemorySwap() int64 { return 0 } </ins> func getContainerIP(container *dockertypes.ContainerJSON) string { if container.NetworkSettings != nil { for _, network := range container.NetworkSettings.Networks {

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: ARG BASEIMAGE ARG REGISTRY <ins> ARG OS_VERSION </ins> # We're using a Linux image to unpack the archive, then we're copying it over to Windows. FROM --platform=linux/amd64 alpine:3.6 as prep

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted