Question: <del> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package flowcontrol import ( "fmt" "reflect" "testing" "time" "github.com/google/go-cmp/cmp" flowcontrol "k8s.io/api/flowcontrol/v1beta3" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func Test_configController_generatePatchBytes(t *testing.T) { now := time.Now().UTC() tests := []struct { name      string condition flowcontrol.FlowSchemaCondition want      []byte }{ { name: "check if only condition is parsed", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            "test none", LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has double quotes", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            `test ""none`, LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ""none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has a whitespace character that can be escaped", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            "test u0009u0009none", LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ttnone"}]}}`, now.Format(time.RFC3339))), }, { name: "check when a few fields (message & lastTransitionTime) are missing", condition: flowcontrol.FlowSchemaCondition{ Type:   flowcontrol.FlowSchemaConditionDangling, Status: flowcontrol.ConditionTrue, Reason: "test reason", }, want: []byte(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":null,"reason":"test reason"}]}}`), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got, _ := makeFlowSchemaConditionPatch(tt.condition) if !reflect.DeepEqual(got, tt.want) { t.Errorf("makeFlowSchemaConditionPatch() got = %s, want %s; diff is %s", got, tt.want, cmp.Diff(tt.want, got)) } }) } } </del>

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: return visitErr } <del> if o.Attach { </del> <ins> if o.Attach && len(containerName) > 0 { </ins> opts := &attach.AttachOptions{ StreamOptions: exec.StreamOptions{ IOStreams: o.IOStreams,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) err := config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { <del> framework.Failf("failed dialing endpoint, %v", err) </del> <ins> framework.Failf("failed dialing endpoint (initial), %v", err) </ins> } <ins> ginkgo.By("Deleting a pod which, will be replaced with a new endpoint") </ins> config.DeleteNetProxyPod() <del> ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) </del> <ins> ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP) (endpoint recovery)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) </ins> err = config.DialFromTestContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, config.MaxTries, config.EndpointHostnames()) if err != nil { <del> framework.Failf("failed dialing endpoint, %v", err) </del> <ins> framework.Failf("failed dialing endpoint (recovery), %v", err) </ins> } })

What happened: final passing output: What you expected to happen: Probably if this test passes, it should be showing several real endpoints. I don't know exactly what it does, but im assuming that the output could be clarified a little. How to reproduce it (as minimally and precisely as possible): run and look at the logs. Anything else we need to know?: Environment: Kubernetes version (use ): 1.20 (sonobuoy)
/sig-network
/assign
/triage accepted
^ an interesting case here was that was but there was no SIG (typo in the command that did). should the label be only if there is label on the ticket? /sig network
the PR that adds triage/accepeted automation () was merged 3 hours ago, but it'll "actually" get rolled out in the next prow bump. Mostly when merges. Thanks for flagging though, appreciate it! :)
sorry, missed the actual question! :woman_facepalming: (need moar coffee!! :coffee: ) I don't think we should require a label for adding the label. In case someone comments with the following, we'd still want it to be valid.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The above test cleans up the By directives and so on, After reading through the test more its clear that every test sets up a scaffold measures a property of that scaffold sometimes deletes something that is recovered in that scaffold (i.e. a pod that is behind a UDP service) So, the thing that was confusing I guess to me is that we validate empty conditions (i.e. no pods available) once we delete a nodePort svc. Meanwhile also noticed that the original work to clean up error reporting needed to cascade over to these tests as well.Have just finished all that, and also updated comments to reflect the Todo around the statistical computation (we have the ability to do extra polling via but we don't actually have any test that leverages this.As a follow on I guess it would be nice if we did have a test that confirmed that pods were load balanced evenly, esp with the new proliferation of different proxy technologies , I.e. a (or something similar


Question: "podSelector": "Selects the pods to which this NetworkPolicy object applies.  The array of ingress rules is applied to any pods selected by this field. Multiple network policies can select the same set of pods.  In this case, the ingress rules for each are combined additively. This field is NOT optional and follows standard label selector semantics. An empty podSelector matches all pods in this namespace.", "ingress":     "List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod OR if the traffic source is the pod's local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default).", "egress":      "List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8", <del> "policyTypes": "List of rule types that the NetworkPolicy relates to. Valid options are Ingress, Egress, or Ingress,Egress. If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8", </del> <ins> "policyTypes": "List of rule types that the NetworkPolicy relates to. Valid options are "Ingress", "Egress", or "Ingress,Egress". If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8", </ins> } func (NetworkPolicySpec) SwaggerDoc() map[string]string {

Is this a BUG REPORT or FEATURE REQUEST?: bug report /kind bug What happened: I was reading the documentation for as defined What you expected to happen: To have clear documentation How to reproduce it (as minimally and precisely as possible): Read the following, as seen in the 1.12 docs: Anything else we need to know?: Nope Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is it a case whereby only can be specified as , or only , or both (note the ) ? If that's the case, do you think the following would make it clearer?
It is confusing to have commas used for English grammar and array element delineation in the same sentence. What about simply this?
sounds like a great fix! Putting out a quick pr with this change now :)


Question: deleted := false timeout := false var lastPod *api.Pod <del> timer := time.After(30 * time.Second) </del> <ins> // The 30s grace period is not an upper-bound of the time it takes to // delete the pod from etcd. timer := time.After(60 * time.Second) </ins> for !deleted && !timeout { select { case event, _ := <-w.ResultChan():

The e2e test is flaky. It often fails with the error in the test suite. Sample failures: Marking it P0 since it frequently blocks the merge queue.
cc
In the flaky test, the pod is deleted with grace-period=30, but there is no guarantee that the pod is going to be deleted from etcd in 30s, thus the flake. This piece of test is copied from the pod e2e test: In that test, the total timeout is actually larger than 30s, because there is some extra logic (< 30s) between the time when the deletion is requested, and when the timer is started. Given that there is no flake reported for test in , I will fix this flake by extending the timeout to 60s.
This flake should have the same root cause as .
ETA on fix? This is the most frequent cause of test flakes and submit queue blockage.
sent out as a temporary fix.
Hmm, happened in: Which looks like we're just not handling the watch cancellation properly in the test.
does the timeout really fix the watch cancellation? (didn't look too deep at the pr)
I think so. see bprashanth's last comment, the timeout occurs again.
As of 3:20pm Apr.8, last failure occurs at 3:13:18pm, Apr. 7, Mike's fix is merged at 15:04:42, Apr. 7.


Question: // restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") <del> postRestartRunningPods := waitForPods(f, numAllPods, recoverTimeout) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) </del> <ins> for start := time.Now(); time.Since(start) < startTimeout; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPods(f, numAllPods, recoverTimeout) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) } </ins> } }) })

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze


Question: -o "${tmp_kube_master_certs}"  http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-master-certs # Convert the yaml format file into a shell-style file. <del> eval "$(python3 -c ''' </del> <ins> eval "$(${PYTHON} -c ''' </ins> import pipes,sys,yaml <del> items = yaml.load(sys.stdin, Loader=yaml.BaseLoader).items() </del> <ins> # check version of python and call methods appropriate for that version if sys.version_info[0] < 3: items = yaml.load(sys.stdin).iteritems() else: items = yaml.load(sys.stdin, Loader=yaml.BaseLoader).items() </ins> for k, v in items: print("readonly {var}={value}".format(var=k, value=pipes.quote(str(v)))) ''' < "${tmp_kube_master_certs}" > "${KUBE_HOME}/kube-master-certs")"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gce-device-plugin-gpu Cluster up Apr 23 8:30PM PDT Taken from: From a , there are supposed to be three worker/minion nodes set up as well. Started failing after -- so maybe related to ? /cc /sig testing More examples of failing runs: /priority critical-urgent (this is a job on the release/master-blocking board)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It uses image gke-1134-gke-rc5-cos-69--138-0-v190320-pre-nvda-gpu as node image kubekins-e2e image is built from Base image is bootstrap image is built from with debian:buster and install python3.
The node image gke-1134-gke-rc5-cos-69--138-0-v190320-pre-nvda-gpu has no python3. We need to upgrade it.
Thanks for catching this and sending revert. Separately, we should definitely upgrade that image, cos-69 is super old and not supported anymore: /cc regarding GPU image


Question: <del> /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package clock import ( "time" clocks "k8s.io/utils/clock" testclocks "k8s.io/utils/clock/testing" ) // PassiveClock allows for injecting fake or real clocks into code // that needs to read the current time but does not support scheduling // activity in the future. // // Deprecated: Use k8s.io/utils/clock.PassiveClock instead. type PassiveClock = clocks.PassiveClock // Clock allows for injecting fake or real clocks into code that // needs to do arbitrary things based on time. // // Deprecated: Use k8s.io/utils/clock.WithTickerAndDelayedExecution instead. type Clock = clocks.WithTickerAndDelayedExecution // Deprecated: Use k8s.io/utils/clock.RealClock instead. type RealClock = clocks.RealClock // FakePassiveClock implements PassiveClock, but returns an arbitrary time. // // Deprecated: Use k8s.io/utils/clock/testing.FakePassiveClock instead. type FakePassiveClock = testclocks.FakePassiveClock // FakeClock implements Clock, but returns an arbitrary time. // // Deprecated: Use k8s.io/utils/clock/testing.FakeClock instead. type FakeClock = testclocks.FakeClock // NewFakePassiveClock returns a new FakePassiveClock. // // Deprecated: Use k8s.io/utils/clock/testing.NewFakePassiveClock instead. func NewFakePassiveClock(t time.Time) *testclocks.FakePassiveClock { return testclocks.NewFakePassiveClock(t) } // NewFakeClock returns a new FakeClock. // // Deprecated: Use k8s.io/utils/clock/testing.NewFakeClock instead. func NewFakeClock(t time.Time) *testclocks.FakeClock { return testclocks.NewFakeClock(t) } // IntervalClock implements Clock, but each invocation of Now steps // the clock forward the specified duration. // // WARNING: most of the Clock methods just `panic`; // only PassiveClock is honestly implemented. // The alternative, SimpleIntervalClock, has only the // PassiveClock methods. // // Deprecated: Use k8s.io/utils/clock/testing.SimpleIntervalClock instead. type IntervalClock = testclocks.IntervalClock // Timer allows for injecting fake or real timers into code that // needs to do arbitrary things based on time. // // Deprecated: Use k8s.io/utils/clock.Timer instead. type Timer = clocks.Timer // Ticker defines the Ticker interface. // // Deprecated: Use k8s.io/utils/clock.Ticker instead. type Ticker = clocks.Ticker </del>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I discovered a need to update one of the two clock packages, and preparation for that revealed the divergent history of the two. One line of development can be seen in , and then It looks like early on that was cloned into , and from there we have an independent line of development: What you expected to happen: The two clock packages would be kept in sync for a while and then one removed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Whoops! Thanks for noticing! Tagging recent other editors: Which one should we keep?
I think the one in is necessary, because it is consumed by component-base, on which apimachinery depends. I created to start fixing the clock package in utils.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/assign
- once and are merged in, the codebase will be fully migrated away from . How can we proceed with removal of this package? Is there some policy around this? /assign
^^ [I would personally vote for just removing it - maybe just leaving some "doc" file to point to the new location of that. But let's maybe wait for others.]
There seem to be a lot of out-of-tree uses of it ([lang][0]=Go) ... if it's easy to use type aliases to map to matching interfaces/implementations and mark the versions deprecated for a release or two, that would ease people updating library versions If that doesn't work or makes the package unmaintainable in some way, removing with a docs pointer seems ok.

Hey Before making any more changes, I wanted to get your opinion on where we go from here: Following , since we cannot modify the interface anymore, one obvious approach that I can think of would be to introduce a new interface in And in (the one to deprecate) Does this sound OK?
Friendly ping ^ :)
I don't have strong opinions about the transition in using a matching library version there is expected I'll defer to on the plan for the utils interface... defining a new interface is the only real way I see to stay backward compatible with the existing exposed go API there. To avoid repeating issues of the past, settle on the final/full interface you need to collapse onto before merging to utils
how does this sound to you wrt unblocking the migration?
This sounds good to me. Given that we migrated everything finally, I think this is ok.
Thanks Based on the above, I tested out how the migration would look locally by making point to my local clone which has and it looks as follows: Could you take a look whenever you get a chance and confirm if this is how we want to proceed? I just want to make sure that we are completely sure before making any more changes :)
Commented there
Hooray! Thanks, !


Question: protocol := utiliptables.ProtocolIpv4 if net.ParseIP(config.BindAddress).To4() == nil { <ins> glog.V(0).Infof("IPv6 bind address (%s), assume IPv6 operation", config.BindAddress) </ins> protocol = utiliptables.ProtocolIpv6 }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Upon code inspection, found that kube-proxy server takes into consideration a bind address of 0.0.0.0 (IPv4), but does not check for a bind address of :: (all-zeros IPv6) when calculating its node IP. What you expected to happen: IPv6 address "::" should be taken into consideration when calculating node IP. How to reproduce it (as minimally and precisely as possible): Code inspection. Anything else we need to know?: Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /sig network


Question: import ( "bytes" "fmt" <ins> "io/ioutil" </ins> "net" <ins> "regexp" </ins> "strconv" "strings" "sync"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Deploying kube-proxy with mode ipvs on a system with the required ipvs support compiled into the kernel rather than as modules causes the kube-proxy deployment to fall back to using iptables. tryIPVSProxy is only checking for the presence of kernel modules and not if the running kernel has the required support compiled in. What you expected to happen: tryIPVSProxy should succeed if either the required modules are found or support is compiled into the running kernel. How to reproduce it (as minimally and precisely as possible): Boot a system using a kernel with ipvs support compiled in rather than as a module (such as the kernel used by and a host without iptables installed Add --config=/etc/kubernetes/kubelet- to the kubelet arguments using the following kubelet- contents: Install using kubeadm with the following config: Check the output of the kube-proxy pod: Anything else we need to know?: From , it should be possible to check for the existence of modules.builtin and if present use that in addition to using modprobe and the contents of /proc/modules. Environment: Kubernetes version (use ): v1.10.2 Cloud provider or hardware configuration: local kvm host OS (e.g. from /etc/os-release): custom linuxkit build Kernel (e.g. ): Linux linuxkit- 4.14.40-linuxkit SMP Thu May 10 10:06:57 UTC 2018 x86_64 Linux Install tools: kubeadm
/sig network
/cc
cc
cc please take a look.


Question: # The manifest command is still experimental as of Docker 18.09.3 export DOCKER_CLI_EXPERIMENTAL=enabled <del> TRIPLE.windows-amd64 := x86_64-w64-mingw32.static </del> <ins> TRIPLE.windows-amd64 := x86_64-w64-mingw32 </ins> TRIPLE.linux-amd64 := x86_64-linux-gnu TRIPLE.linux-arm := arm-linux-gnueabihf TRIPLE.linux-arm64 := aarch64-linux-gnu

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When including windows in a pause build I get a rate limit error as it tries to pull dockcross/windows-static-x64:latest. This causes the build to fail. This image should be stored and pulled from as there should be a copy of this in to mimic kube-cross for Linux images cd build/pause make all-push Kubernetes version : v1.20.1 Cloud provider or hardware configuration: N/A OS (e.g: ): Build running on Linux (includes cross compiling for windows) Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Are you either of you able help with getting this image into the k8s gcr or if there is a reason why it can't be put there??
context - this might be due to use pulling from a small # of vmware VIPs
goal from sig-windows ~ move this into GCR eventually ~ would that be ok ?
do you know any history for this image?
That image carried over from the original work from . I have asked sig-testing previously if we could include that image's functionality into ` into the image. I can try this out in the morning, and if it works, I can send some PRs.
Seems to work. I have sent 2 PRs for this:


Question: "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" "k8s.io/client-go/tools/cache" <ins> "k8s.io/client-go/tools/record" </ins> "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/controller/history" )

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: package factory import ( <ins> "time" </ins> "github.com/coreos/etcd/clientv3" "github.com/coreos/etcd/pkg/transport" "golang.org/x/net/context"

Hi, running 1.6.4 on etcd 3.0.17 in the following setup: 3 etcd nodes (bootstrapped according to the ) 9 kubernetes cluster nodes all of them running kube-apiserver (--apiserver-count=9) with --bind-address to local kube-apiserver instance. All instances pointing to the same etcd cluster (--etcd-servers=http://IP1:4001,http://IP2:4001,http://IP3:4001) I tried to simulate 1 etcd node crash by simply turing off one of the machines. Result: timeouts running kubectl (get nodes, get pods) commands lots of these error messages in kube-apiserver logs: "] apiserver received an error that is not an metav1.Status: http: Handler timeout" ETCD correctly reports unhealthy node, but healthy cluster Inspecting netstat shows lots of ESTABLISHED connections to the dead etcd server, port 4001 - these connections are cleared only by restarting the kube-apiserver(s). Workarounds: Switching back to etcd 2.2.1 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds Leaving etcd 3.0.17 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds In both above cases kubectl shows expected results. The dead node is evicted (NotReady), other nodes & pods are in tact. I noticed that client contains TCP KeepAlive timeout, while client does not. Could this be related to the described scenario? <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why.

--Kubernetes version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: OS: NAME="Oracle Linux Server" VERSION="6.7" ID="ol" VERSIONID="6.7" PRETTYNAME="Oracle Linux Server 6.7" ANSICOLOR="0;31" CPENAME="cpe:/o:oracle:linux:6:7" HOMEURL="" BUGREPORTURL="" ORACLEBUGZILLAPRODUCT="Oracle Linux 6" ORACLEBUGZILLAPRODUCTVERSION=6.7 ORACLESUPPORTPRODUCT="Oracle Linux" ORACLESUPPORTPRODUCTVERSION=6.7 Kernel
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/area etcd
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to depend on an issue with etcd, there is no option for a KeepAlive I believe. I've sent a PR, can fix this once that gets merged.
The mentioned PR has been merged. We may need to wait for an etcd release / update the dependency on etcd for us to be able to fix this in Kubernetes.
Had a look at the current CHANGELOG on etcd. The fix for this issue is in master, and will be released in etcd clientv3 3.3.0.

Here is the which shows the fix has been merged into the . Kubernetes is currently on clientv3 3.2.11 it appears (according to ).


Question: }, "time": { "$ref": "#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.Time", <del> "description": "Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'" </del> <ins> "description": "Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over." </ins> } }, "type": "object"

Godoc says that Time should be empty on mangedFields entry created by "Apply" operation, but kubernetes populates time on apply. Godoc consistent with kubernetes behaviour Godoc: Kubernetes: No response <details</details<details</details<details</details<details</details<details</details<details</details
/wg api-expression /sig api-machinery
/cc
/cc /triage accepted
Yeah, good catch. The comment matches the behavior from alpha, which we changed later on. Thanks for letting us now :-) I think we could do a better job describing how this value is changed now.


Question: } w.containerID = kubecontainer.ParseContainerID(c.ContainerID) w.resultsManager.Set(w.containerID, w.initialValue, w.pod) <ins> // We've got a new container; resume probing. w.onHold = false } if w.onHold { // Worker is on hold until there is a new container. return true </ins> } if c.State.Running == nil {

Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !


Question: defer proxier.mu.Unlock() proxier.servicesSynced = true } <ins> func deleteServices(proxier *Proxier, allServices ...*v1.Service) { for i := range allServices { proxier.OnServiceDelete(allServices[i]) } proxier.mu.Lock() defer proxier.mu.Unlock() proxier.servicesSynced = true } </ins> func makeTestService(namespace, name string, svcFunc func(*v1.Service)) *v1.Service { svc := &v1.Service{ ObjectMeta: metav1.ObjectMeta{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Two services referring the same endpoint. One of the services was deleted. This brought down the connectivity of the other service. What you expected to happen: The connectivity to service should continue to work How to reproduce it (as minimally and precisely as possible): two services referencing same endpoint one of the services connectivity is broken to the other service Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Do you mean a service referencing the same pod, or literally the same Endpoints object?
I meant service referencing the same pod.
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Is this still an issue? If so, it would be helpful to get an answer to Jordan's question above. If this Service is referencing the same Endpoints object that would be a very different issue than if it was referencing the same Pod. /triage needs-information
This issue happens when multiple Services are referencing a same POD. I mean the same backend IP address but different ports. i.e. the POD is hosting two workloads on different ports.
/sig windows
Is this only observed in windows environments?
Yes this happens only on Windows environments
Thanks can you provide more information about the version of Kubernetes you're using? Do you know what proxier mode kube-proxy is running with? Maybe winkernel? I'm not as familiar with the Windows implementation here, adding to help debug this.


Question: newJobSchedule string } tests := []struct { <del> name                 string fields               fields args                 args deltaTimeForQueue    time.Duration roundOffTimeDuration time.Duration </del> <ins> name              string fields            fields args              args deltaTimeForQueue time.Duration </ins> }{ { name: "spec.template changed", fields: fields{ <del> queue:          workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "test-update-cronjob"), </del> <ins> queue:          &fakeQueue{RateLimitingInterface: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "test-update-cronjob")}, </ins> recorder:       record.NewFakeRecorder(10), jobControl:     jc, cronJobControl: cjc,

pull-kubernetes-unit TestController2updateCronJob/spec.schedulechanged TestController2_updateCronJob /sig apps /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is important to resolve before promotion to v1
looks like the cause was
This function should be related.
Another one:
When the flake happen, there are messages like When the system is overloaded, it would happen. is trying to replace the current queue in tests with a mock.
Yeah, I spoke with to make sure we use a mock of the queue so we only get the most important bit of information ignoring the remaining parts


Question: var nl *NoOpLegacyHost pluginSettings.LegacyRuntimeHost = nl <ins> if containerRuntime == kubetypes.RktContainerRuntime { glog.Warningln("rktnetes has been deprecated in favor of rktlet. Please see https://github.com/kubernetes-incubator/rktlet for more information.") } </ins> // rktnetes cannot be run with CRI. if containerRuntime != kubetypes.RktContainerRuntime { // kubelet defers to the runtime shim to setup networking. Setting

rktnetes is the old, in-tree rkt integration. is new integration via CRI. We should deprecate and remove the rktnetes integration once rktlet is ready to take over. Based on the , I think we can announce the deprecation of rktnetes once rktlet is ready, and remove the integration after one more release. /cc who have been working on rktlet who are the owners of rktnetes
cc who once asked me about this confusion and I suggested him to do a clean up. _<
Thanks, very glad to do the clean up part. Is there any plan for deprecation part?
It depends on the progress of rktlet. The folks who are currently working on rktlet and are migrating off rktnetes would be a better source.
It was included in sig-node's 1.9 planning draft, and discussed at sig-node meeting. It was removed for 1.9 because 1) the user (for example: blablacar) is currently depending on it 2) rktlet is under development. We can announce the deprecation now to prevent more users from using it. Once rktlet is done, we can remove it completely.
We're currently working towards an initial rktlet release, which will be tested by Blablacar. We'll give an update next week with the progress.
what's the current progress of the rktlet release?
Hi sorry for the late reply. We plan on releasing rktlet 0.1.0 this week. In that release we implement basic support for the CRI including fetching images, running pods, CNI networking, logging and exec. We pass 129/145 e2e conformance tests. Some features (e.g. attach, port forward) are not implemented so we don't yet consider it ready for deprecating rktnetes. Those features were out of scope for the work we were contracted to do so I’m not sure when it will be completed.
thanks for the update! I don't think rktnetes ever supports attach or port forward, so I don't think they should be the blocker for deprecating rktnetes. Adding to confirm.

(rktlet maintainers), I'd like to aim at officially deprecating rktnetes in k8s v1.10 (and encourage users to move to rktlet). Do you think this is reasonable? We can discuss more in the sig node meeting or on slack if you have concerns.
From the today's sig-node meeting, current rktlet maintainers ( etc) mentioned that they may not have spare cycles to work on rktlet this release, but even as of today, rktlet already passes more tests than rktnetes. Given that, I propose we deprecate rktnetes in 1.10, and remove the code in the next release. The deprecation process includes Send out a deprecation announcement to kubernetes-users kubernetes-dev and kubernetes-sig-node@ with links to rktlet. Add deprecation warning to kubelet when --container-runtime is set to rkt. Can someone from the CoreOS (owner of rktnetes and initiator of rktlet) comment on whether the plan is okay?
/cc
LGTM, do you need any help from our side?
I haven't been involved in active rkt development for a while, but this plan sounds fine to me based on everything I know about the latest rktnetes work.
The plan sounds good to me. The only concern I have is whether Blablacar needs rktnetes for a longer time (cc
I'd love to hear Blablacar's feedback on this. At the same time, I think it might be reasonable for Blablacar not to upgrade kubernetes until they are ready. There hasn't been much active development in rktnetes, so staying on a known, good version, is better than upgrading kubernetes anyway IMO.
I think the rktnetes maintainers are the best candidate to send out the deprecation announcement :-) Adding deprecation warning is trivial. If anyone wants to help, that'd be great.
Sorry for the delay, I was off last week. We (BlaBlaCar) are moving away from rktnetes implementation so it's ok for us if it gets deprecated. Thanks for asking :)
has sent out the announcement. Will close this once the PR gets merged.
Seems like is dead and does not support v1.10.


Question: controllerName := "bootstrap-controller" coreClient := corev1client.NewForConfigOrDie(c.GenericConfig.LoopbackClientConfig) <del> bootstrapController := c.NewBootstrapController(legacyRESTStorage, coreClient, coreClient, coreClient, coreClient.RESTClient()) </del> <ins> bootstrapController, err := c.NewBootstrapController(legacyRESTStorage, coreClient, coreClient, coreClient, coreClient.RESTClient()) if err != nil { return fmt.Errorf("error creating bootstrap controller: %v", err) } </ins> m.GenericAPIServer.AddPostStartHookOrDie(controllerName, bootstrapController.PostStartHook) m.GenericAPIServer.AddPreShutdownHookOrDie(controllerName, bootstrapController.PreShutdownHook)

Clusters created with kubeadm using DualStack are not able to work, because CoreDNS pods never get ready Seeing one failed job logs It seems that the pod is "waiting for the kubernetes service"?? coredns version seems the same as in the working jobs CoreDNS pod should be ready Instal a Dual-stack cluster using master with kubeadm (or kind ) ! It started to fail on Oct12 <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc I didn't notice it was failing for more than 1 month :facewithhead_bandage: I didn't have time to dig into it yet, but this PR landed just that day how is that the pods running has the version 1.8.4 but the PR seems to update kubeadm to 1.8.6?
Looking into this today. The error from CoreDNS should only happen when the kubernetes plugin fails to sync with the Kubernetes API, either because it can’t connect, or it can’t sync the lists of services/endpointslices. There were changes and fixes to the API connection code, so it could be a regression.
Actually no. The changes I was thinking of were already in 1.8.4.
How can we reproduce it : Instal a Dual-stack cluster using master with kubeadm (or kind ) I was unable to reproduce this issue in a kind dual stack cluster.

After upgrading coredns to 1.8.6 (from 1.8.0 installed by kind), and adding list/watch permission for endpointslice, the pods were ready. edit: It also works for coredns 1.8.4 - I'm unclear as to which version I should be checking.
let me check if is a regression in kind :thinking:
working with 1.8.6 and yesterdays kubernetes podIPs
This is weird, I hit a similar issue in openshift, endpoints doesn't match the IP family of the kubernetes.default service apiserver service is single stack, at least until this KEP fixes it What I don't understand is why the order of the IPs in kind is different let me try to fix both things /assign
my fault, sorry for the noise :sweat_smile:


Question: for _, v := range vmMoList[0].Guest.Net { if vs.cfg.Network.PublicNetwork == v.Network { for _, ip := range v.IpAddress { <del> if net.ParseIP(ip).To4() != nil { </del> <ins> if !net.ParseIP(ip).IsLinkLocalUnicast() { </ins> nodehelpers.AddToNodeAddresses(&addrs, v1.NodeAddress{ Type:    v1.NodeExternalIP,

What happened: IP addresses losing in k8s node status in a pure ipv6 vsphere environment. all interfaces of nodes in this cluster are ipv6 only The error logs in kubelet as the following: What you expected to happen: It will be very appreciated if we can get ipv6 addresses from vsphere cloud provider as ipv4 env How to reproduce it (as minimally and precisely as possible): deploy k8s cluster in vsphere pure ipv6 envirenment Anything else we need to know?: From the code we can know that while the provider retrieves VM’s IP, It uses To4() function for all ip addresses from vsphere platform, this function can’t handle ipv6 addresses as mentioned in guild: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g: ):Kernel (e.g. ):Install tools: ansible Network plugin and version (if this is a network-related bug): calico v3.5.6 Others:
If you agree with this I can fix it.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
cc /remove-sig contributor-experience
/cc
please help to review, I think I can fix the issue by PR /sig cloud-provider Thanks a lot
/assign


Question: import ( "fmt" <del> "reflect" </del> <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" unversionedvalidation "k8s.io/apimachinery/pkg/apis/meta/v1/validation" "k8s.io/apimachinery/pkg/labels"

/kind bug When deploying a StatefulSet, I included a PVC that specified for the storage capacity. worked perfectly. Running again without making any changes failed with the error: After comparing the server-side yaml to my original, I noticed that it had changed the storage capacity of my PVC to . Changing my original from to allowed me to then successfully run . There is clearly a string representation being used to compare those changes rather than converting to bytes and comparing the numeric representation. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-27T22:29:25Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure AKS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc This is probably an issue for any resource.Quantity type.
/sig api-machinery
the issue is use of in statefulset update validation, instead of : explicitly compares quantity objects semantically:
/sig apps
fix is in with picks open to 1.11, 1.10, 1.9


Question: return vh.kubelet.kubeClient } <ins> // NewWrapperMounter attempts to create a volume mounter // from a volume Spec, pod and volume options. // Returns a new volume Mounter or an error. </ins> func (vh *volumeHost) NewWrapperMounter(volName string, spec volume.Spec, pod *api.Pod, opts volume.VolumeOptions) (volume.Mounter, error) { // The name of wrapper volume is set to "wrapped_{wrapped_volume_name}" wrapperVolumeName := "wrapped_" + volName

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: "type": "string" }, "protocol": { <del> "type": "string" </del> <ins> "type": "string", "nullable": true </ins> } }, "required": [

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, required: []corev1.ResourceName{corev1.ResourceMemory}, <del> err:      `must specify memory`, </del> <ins> err:      `must specify memory for: bar,foo`, </ins> }, "container resource missing": { pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{ <ins> Name: "dummy", Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")}, }, }}, }, }, required: []corev1.ResourceName{corev1.ResourceMemory}, err:      `must specify memory for: dummy`, }, "multiple container resource missing": { pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{ Name: "foo", Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")}, }, }, { Name: "bar", </ins> Resources: api.ResourceRequirements{ Requests: api.ResourceList{api.ResourceCPU: resource.MustParse("1m")}, Limits:   api.ResourceList{api.ResourceCPU: resource.MustParse("2m")},

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: func verifyPrefixOfSecretResource(t *testing.T, wantPrefix string, test *transformTest) { // implementing this brute force approach instead of fancy channel notification to avoid test specific code in prod. // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefix, test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefix, "", test) </ins> // run storage migration secretsList, err := test.restClient.CoreV1().Secrets("").List(

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } else { volumePluginDir = gciVolumePluginDirLegacy } <del> } else if node != nil && framework.NodeOSDistroIs("gci") { </del> <ins> } else if node != nil && framework.NodeOSDistroIs("gci", "ubuntu") { </ins> if getNodeVersion(node).AtLeast(versionutil.MustParseGeneric(gciVolumePluginDirVersion)) { volumePluginDir = gciVolumePluginDir } else {

https://k8s- https://k8s- It passes on COS. /cc
I can look into this, since likely this was introduced because of latest changes I made.
This looks like a timing thing - because I can see message in logs , but I think 5 minutes is not enough for the pod to come up after installing flexvolume.
That's strange, the dummy driver just sets up a tmpfs so it should be fairly quick. I wonder if there's something else blocking pod creation?
the driver is not being installed on kubelet I am doing some more investigating, have my own cluster up.
On ubuntu, we set the flexvolume install dir to --volume-plugin-dir=/home/kubernetes/fl exvolume Small oversight, I'll make a PR shortly
They are still failing, I am having another look e: Same as before, it is not installing on the node. I started my test cluster using (basically...) the same kubetest command sooo allow me to try again. e2: it's a false alarm. The tests are not using a commit with my fix in it yet zzzz


Question: }, }, expectedUpdateCalls: []fakecloud.UpdateBalancerCall{ <ins> {Service: etpLocalservice1, Hosts: []*v1.Node{node1, node2, node3}}, {Service: etpLocalservice2, Hosts: []*v1.Node{node1, node2, node3}}, </ins> {Service: service3, Hosts: []*v1.Node{node1, node2, node3}}, }, }, {

In 1.26 the service controller in the Kubernetes cloud controller manage (KCCM) introduced a new mechanism for syncing load balancers based on predicates. The predicates applied will depend on "service classes", specifically if it's or , see: This implementation is not well suited for internal GCP load balancers which use InstanceGroups. from Google, feel free to correct me: any cluster has all nodes grouped together into an instance group. Service of have individual GCP load balancers, but these load balancers end up referencing the same instance group (which is the cluster's instance group). This means that the inclusion/exclusion predicates for any service, will trickle down and impact the cluster's global instance group, thereby impacting other services which don't necessarily apply the same set of predicates. Ex: services exclude nodes, whereas do not. A Node going will therefore be removed from all backends for services, however on GCP with load balancer solution based on InstanceGroups: this means that the same node will be removed from all backends for services too. This is NOT a problem on = 1.27 using the feature , because all services on 1.27 using that feature gate will have the same set of predicates. Node updates requiring the removal of the instance from load balancers belonging to services of should not impact other services of . Bring up a cluster using "Internal load balancers" on GCP. Define two services, one being and the other . Have one node go . The instance corresponding to that Node might be removed from the InstanceGroup backend of the service. This will depend on the order the services are synced, if the service is synced last then the instance will be removed. This is actually not really that bad, because the only different predicate between both is the readiness condition - and a node should go back to if all is fine, and finally be included. But I wanted to raise this for issue for completeness sake so that it is known. <details</details<detailsGCP </details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/triage accepted


Question: if prevItem == nil { return nil, admission.NewForbidden(a, fmt.Errorf("unable to get previous usage since prior version of object was not found")) } <del> prevUsage := evaluator.Usage(prevItem) deltaUsage = quota.Subtract(deltaUsage, prevUsage) </del> <ins> // if we can definitively determine that this is not a case of "create on update", // then charge based on the delta.  Otherwise, bill the maximum metadata, err := meta.Accessor(prevItem) if err == nil && len(metadata.GetResourceVersion()) > 0 { prevUsage := evaluator.Usage(prevItem) deltaUsage = quota.Subtract(deltaUsage, prevUsage) } </ins> } if quota.IsZero(deltaUsage) { return quotas, nil

With the migration to podutils and adding some presubmits (from periodic jobs) we are seeing some failures in presubmits. The error seems to have issues finding the right node to create.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig testing
any leads on solving this one?
see notes on slack -
/assign
this looks good now!


Question: }, }, { <ins> name: "handles empty object name", infos: []*resource.Info{ { Mapping: &meta.RESTMapping{ Resource: schema.GroupVersionResource{Group: "group", Version: "version", Resource: "theresource"}, }, Namespace: "ns-foo", }, }, fakeClient: func() *dynamicfakeclient.FakeDynamicClient { return dynamicfakeclient.NewSimpleDynamicClient(scheme) }, timeout:     10 * time.Second, expectedErr: "resource name must be provided", validateActions: func(t *testing.T, actions []clienttesting.Action) { if len(actions) != 0 { t.Fatal(spew.Sdump(actions)) } }, }, { </ins> name: "times out", infos: []*resource.Info{ {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a job with metadata.generateName twice and waited for completion, but got: panic: interface conversion: runtime.Object is unstructured.UnstructuredList, not unstructured.Unstructured What you expected to happen: kubectl to wait for the last job to complete. How to reproduce it (as minimally and precisely as possible): kubectl create -f kubectl create -f kubectl wait --for=condition=complete -f (shortened): apiVersion: batch/v1 kind: Job metadata: generateName: my-job- Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have to choose a sig, but I am not sure which one is right...
Reiterating the mentions to trigger a notification: <detailsIn response to : I have to choose a sig, but I am not sure which one is right... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli /sig apps
/assign
kubectl get -f also doesn't work as expected.
/remove-sig apps


Question: // balancers (l4) for performing health checks. func MakeHealthCheckFirewallName(clusterID, hcName string, isNodesHealthCheck bool) string { if isNodesHealthCheck { <del> return makeNodesHealthCheckName(clusterID) + "-http-hc" </del> <ins> return MakeNodesHealthCheckName(clusterID) + "-http-hc" </ins> } return "k8s-" + hcName + "-http-hc" } <del> func makeFirewallName(name string) string { </del> <ins> // MakeFirewallName returns the firewall name used by the GCE load // balancers (l4) for serving traffic. func MakeFirewallName(name string) string { </ins> return fmt.Sprintf("k8s-fw-%s", name) }

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: type unsetConfigTest struct { description string <del> config      clientcmdapi.Config //initiate kubectl config args        []string            //kubectl config unset args expected    string              //expect out </del> <ins> config      clientcmdapi.Config args        []string expected    string expectedErr string </ins> } func TestUnsetConfigString(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5.3 Environment: Cloud provider or hardware configuration: minikube 0.17.1 How to reproduce it (as minimally and precisely as possible): config Note that there's only one context. the property of a non-existent context config Note that a context named foo was created just because I tried to unset a property. What you expected to happen: should give an informative message saying that context doesn't exist. Anything else we need to know: This is probably low-hanging fruit.
/sig cli
/assign i will take a look for this
GitHub didn't allow me to assign the following users: a, look, for, this, i, will, take. Note that only can be assigned. <detailsIn response to : /assign i will take a look for this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can reproduce this, this is a bug, have open PR fix this. thanks
/kind bug


Question: # See https://github.com/golang/go/issues/16086 for details. go install ./cmd/... <del> # Use eval to preserve embedded quoted strings. eval "goflags=(${GOFLAGS:-})" </del> # Filter out arguments that start with "-" and move them to goflags. targets=() for arg; do

What happened: When building kubernetes from source for 1.10 it was possible to limit the parallel linker calls using GOFLAGS="-p 1". (I understand that the new format is GOFLAGS="-p=1" so I am using that) With kubernetes 1.11 the build aborts: What you expected to happen: The build should complete, using just one process for linking. How to reproduce it (as minimally and precisely as possible): export GOFLAGS="-p=1" make -j1 Anything else we need to know?: The environment is quite memory constrained, therefore preventing the build to fork a linker for each CPU is required. Any other option instead of the GOFLAGS would work as well. Environment: Kubernetes version (use ): 12.0 OS (e.g. from /etc/os-release): Arch Linux ARM Others: go version go1.11.1 linux/arm64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
This is because a change in go 1.11, see , and related go issues: , etc. I've opened a PR to address this: .
/assign


Question: return body } <ins> // GetHTTPContentFromTestContainer returns the content of the given url by HTTP via a test container. func GetHTTPContentFromTestContainer(config *e2enetwork.NetworkingTestConfig, host string, port int, timeout time.Duration, dialCmd string) (string, error) { var body string pollFn := func() (bool, error) { resp, err := config.GetResponseFromTestContainer("http", dialCmd, host, port) if err != nil || len(resp.Errors) > 0 || len(resp.Responses) == 0 { return false, nil } body = resp.Responses[0] return true, nil } if pollErr := wait.PollImmediate(framework.Poll, timeout, pollFn); pollErr != nil { return "", pollErr } return body, nil } </ins> // DescribeSvc logs the output of kubectl describe svc for the given namespace func DescribeSvc(ns string) { framework.Logf("nOutput of kubectl describe svc:n")

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: pod := StartInPodWithInlineVolume(client, t.Namespace, "inline-volume-tester", command, volumes, t.ReadOnly, t.Node) defer func() { // pod might be nil now. <del> StopPod(client, pod) </del> <ins> StopPodAndDependents(client, pod) </ins> }() framework.ExpectNoError(e2epod.WaitForPodRunningInNamespaceSlow(client, pod.Name, pod.Namespace), "waiting for pod with inline volume") runningPod, err := client.CoreV1().Pods(pod.Namespace).Get(context.TODO(), pod.Name, metav1.GetOptions{})

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: I0722 06:26:18.056] I0722 06:26:18.057] [Driver: csi-hostpath] I0722 06:26:18.057] I0722 06:26:18.057] [Testpattern: Generic Ephemeral-volume (default fs) [Feature:GenericEphemeralVolume] (late-binding)] ephemeral skipped 448 lines unfoldmore I0722 06:30:33.845] Jul 22 06:30:33.845: INFO: Wait up to 5m0s for pod "inline-volume-tester2-hhcbd" to be fully deleted I0722 06:31:06.117] Jul 22 06:31:06.117: INFO: Pod inline-volume-tester-74jqh has the following logs: /dev/sda1 on /mnt/test-0 type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30) I0722 06:31:06.117] I0722 06:31:06.117] Jul 22 06:31:06.117: INFO: Deleting pod "inline-volume-tester-74jqh" in namespace "ephemeral-9141" I0722 06:31:06.155] Jul 22 06:31:06.155: INFO: Wait up to 5m0s for pod "inline-volume-tester-74jqh" to be fully deleted I0722 06:31:38.264] Jul 22 06:31:38.264: FAIL: no dangling PVCs I0722 06:31:38.264] Expected I0722 06:31:38.265] <[]v1.PersistentVolumeClaim | len:1, cap:1: [ I0722 06:31:38.265] { I0722 06:31:38.265] TypeMeta: {Kind: "", APIVersion: ""}, I0722 06:31:38.

265] ObjectMeta: { skipped 315 lines unfold_more I0722 06:32:16.896] STEP: Destroying namespace "ephemeral-9141" for this suite. I0722 06:32:16.931] Jul 22 06:32:16.931: INFO: Namespace ephemeral-9141 was already deleted I0722 06:32:16.932] STEP: Destroying namespace "ephemeral-522" for this suite. I0722 06:32:16.966] Jul 22 06:32:16.966: INFO: Namespace ephemeral-522 was already del Anything else we need to know: /cc /sig storage /priority critical-urgent /milestone v1.19
I have tracked this down and it looks like that changes were made in that affected how Pods were being deleted in the e2e framework (specifically commit ). This caused failures in other tests, so the commit was reverted in I will have a fix open shortly that uses foreground deletion only for these storage tests. /assign /cc Side note: I see already saw this