Question: // The tests assume that Get/GetList/Watch calls shouldn't fail. // However, 429 error can now be returned if watchcache is under initialization. // To avoid rewriting all tests, we wait for watcache to initialize. <del> if err := cacher.ready.wait(context.Background()); err != nil { </del> <ins> if err := cacher.Wait(context.Background()); err != nil { </ins> return nil, storage.APIObjectVersioner{}, err } }

pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.


Question: && tar xzf v1.9.0.tar.gz  && rm -f v1.9.0.tar.gz <ins> RUN python /models-1.9.0/official/wide_deep/data_download.py </ins> WORKDIR $HOME/models-1.9.0/official/wide_deep ENV PYTHONPATH=$PYTHONPATH:$HOME/models-1.9.0 <del> ENTRYPOINT python ./data_download.py && python ./wide_deep.py </del> <ins> ENTRYPOINT python ./wide_deep.py </ins>

E2eNode Suite.[sig-node] Node Performance Testing [Serial] [Slow] Run node performance testing with pre-defined workloads TensorFlow workload From the testgrid, it has been 1/2 flaky since the start of history (03/20/2022) and has been failing constantly (0/2) since 04/01/2022 No response /sig node
/triage accepted /priority important-soon
The failing test seems to be expecting GPU on machine and the image seems to be broken. I tried the image manually: Logs:
/assign as discussed at SIG Node CI meeting, can you please take a look? I see that pod is now running for couple minutes and then stops. I wonder if this is something environment related - maybe pod was evicted or something. I doesn't seem to be able to repro the failure of the pod locally
GitHub didn't allow me to assign the following users: xmcqueen. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is no longer a flake. It has failed every run since the start of June. Locally this test runs perfectly with a legit correct local run of the test suite code, doing the full tensorflow run of 300 epochs. The failure in the testgrid job seems to be fast (173 sec to build and fail, locally its 160 sec before it starts the tensorflow pod). I wonder if there's a blocked port or something from the data initializing python program. It needs access to: . Here's the command it runs: Is it feasible to add debugging code, like a pre-check for that http port? It would be good to at least force it to fail with a meaningful error message.
Note that HEAD is now at image version 1.2, and it is still failing per . The successful local runs of the test mentioned above used image version 1.2.

Locally image version 1.1 does produce the cuda/GPU error shown above by :
More info is needed to find the RC for this test failure. I've put in a PR to get the test framework to preserve the container logs when the pod fails.
The pod logs show that it is timing out on an http connection, and it looks like it's the one where it tries to download the test data set for the tensorflow run. You can see it . Search down to . The data_download script in the image attempts to pull down a dataset from: . That url is live and healthy, so they might be blocking the gcloud.
This test depending on that external data doesn't look quite right :/
I'm discussing this test with sig-testing. I'll post an update later.
Talked to from sig-testing: he's contacting the remote to see if they are blocking us, because it looks like they are is pulling the script from a that is So I see three choices: upgrade this thing to a more recent edition delete it grab everything that's needed to run this and keep it in-house <- clear winner here what do you think: Maybe the folks over at sig-testing have a place to store the artifacts in-house? They are not massive (about 10MB see below) Size of data about 10MB: the tools and libs for the tensorflow run: the training and eval files for the testrun:
Here's a relevant issue in a related k8s repo about using/not using docker registries: Registries used in k/k CI should be on Kubernetes Community infra
looks good for the subject of putting up a new docker image containing the deps. This seems feasible and effective. I'd prefer it to deleting the test. I'll discuss it a bit more with a few others, but this seems like the way to go forward with this. TBS, I plan to take the above listed test requirements, and build a docker image containing all that is needed, and reset the test to pull the new docker image.
I've got some Dockerfile changes to fix this. I've tested it locally with no network and it does the full 300 epochs. PR coming soon.
Is it worth pushing up 182 files to get this green? It is about 6M of python, and 5M of a test dataset.

Its all stashed away in the to get the docker image to contain the prereqs. It is selected subdirectories of the quite old archived version (v1.9.0) of the code from here: current version v2.7.2 and two small test data sets from here:
OK I got it down to 43 files, including the 2 datasets. That's the drivers and the datasets and the libs they need. It all gets packed into the docker image.
quick question - is there another repository that is under sig-node that can host this image?
To be clear, the test's docker image is currently in a repo, the : This PR adds artifacts to an existing docker image in the above repo.
I've cancelled the above PR. I'll try to get something else going.
There has been a decent amount of discussion in slack relating to handling the test's required artifacts. The solution that seems best is to get some practical version of the test requirements into locally owned and controlled storage, so the endpoints that provide access to the assets will at least be under internal control. This is not code, and is not under development. It is a static dataset, that is now stored in an offsite archive, and will not require maintenance. The datasets are published with the "": (see bottom right) "This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given." Here are some links to the dataset and where its hosted: https://archive-
I'd like to get the dataset that holds the test data (about 650k) hosted in GCS where I can get it via http. With that change it is expected that the test will go green. The failures show it is failing when it tries to download the artifact from the uncontrolled remote endpoint. Somebody, please help me get this artifact into an internally controlled server. Thanks
Was the idea of downloading this only on image creation in dockerfile rejected?
Thanks for the reminder I did not fairly evaluate that proposal. I just jumped to my earlier proposal. Thinking about this proposal I see that it would mean downloading the test data for inclusion into the docker image, while the image's test driver tooling is built to pull down the dataset itself from a remote.

That is a simple change to handle, while simply re-hosting the artifact on GCS would require only changing the endpoint. Also building the docker would probably still require network access to the artifact, which would probably require us to put it somewhere anyway. I assume, putting it into the docker image would best be done by downloading the test data artifact from somewhere. Would we then be storing the artifact twice - once in the docker image, and once in a locally controlled endpoint? If the docker build server has unrestricted access to the artifact, this proposal would be good. I don't have access to the servers to run tests. However, I could put up an RB for the Dockerfile to try it out. This is probably worth the effort. I'll get an PR going for trying to pull down the remote test-data artifact from the image build server, and bunde it into the docker image, and I'll link it to this ticket.
thank you. We do not change the image for a long time, so downloading it once would likely be enough. Once it is self-contained, it is even better for the continuous perf runs in tests.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the image builds for the first arch (amd64) but fails when it enters the cross compiling phase (arm64) and specifically when it tries to do apt installs from within the qemu-*-static emulator: The other images in node-perf fail to build in the same way. The jessie-dnsutils image build fails to build in the same way. I have tried pulling in newer base images up to buster, but it did not fix the problem. -Start focusing on the version of qemu. - qemu is version locked in the Makefile and has not changed in two years. There was an and the images were building at that time.

Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.


Question: pp.timeoutPermit = false pp.waitAndRejectPermit = false pp.waitAndAllowPermit = false <ins> pp.allowPermit = false </ins> } // NewPermitPlugin is the factory for permit plugin.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: // priority level if it has no more use.  Call this after getting a // clue that the given priority level is undesired and idle. func (cfgCtlr *configController) maybeReap(plName string) { <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> plState := cfgCtlr.priorityLevelStates[plName] if plState == nil { klog.V(7).Infof("plName=%s, plState==nil", plName)

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: return nil, err } } <del> sort.Strings(vals) </del> return &Requirement{key: key, operator: op, strValues: vals}, nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet.

If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.


Question: original[k] = patchV continue } <ins> // If the data type is a pointer, resolve the element. if t.Kind() == reflect.Ptr { t = t.Elem() } </ins> // If they're both maps or lists, recurse into the value. // First find the fieldPatchStrategy and fieldPatchMergeKey. fieldType, fieldPatchStrategy, fieldPatchMergeKey, err := forkedjson.LookupPatchMetadata(t, k)

PATCH http://IP:8080/api/v1beta3/namespaces/dev--ea6f-49cb-b5bb-/replicationcontrollers/redis-master {"spec":{"replicas":0}} answer: { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server responded with the status code 415 but did not return more information", "details": {}, "code": 415 } Before v0.15.0 it had worked well.
That error is returned by go-restful because it did not find any valid route with the same content type as specified in the request: Can you try setting the content-type in your request as is described here: cc
well, I've the same issue which is not fixed by setting the value, at least with the "strategic merge patch". Here's the request: Here's the response body: And here's the log trace of the request in the apiserver pod:
Come from a typo, purposed PR here:
Btw, I'm still having strange issues with sub-objects. Note that it appears only is the sub-object is already set. Here's the request: Here's the API log:
Looks like a bug in strategic merge -- it doesn't deal with pointers correctly:
cc
is it possible to have a milestone for this so I can tell if I should merger ? Thanks.


Question: "time" autoscalingv2 "k8s.io/api/autoscaling/v2beta2" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta/testrestmapper" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down:


Question: // Register registers a collectable metric but uses the global registry Register = defaultRegistry.Register <ins> // Registerer exposes the global registerer Registerer = defaultRegistry.Registerer </ins> ) func init() {

At (originally ) component-base metrics exposed Registerer API for raw prometheus registration, and I took dependency in The resulting does not seem to emit the new metrics (example: ). The metrics handler should emit the registered metrics. Falling back to does work. So there is some material difference with . 24, 1.25, 1.26 GKE No response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign jkh52


Question: ### Create POD valid-pod from JSON # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command <del> kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" </del> <ins> kubectl create "${kube_flags[@]}" -f examples/limitrange/valid-pod.json </ins> # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' kube::test::get_object_assert 'pod valid-pod' '{{.id}}' 'valid-pod' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' kube::test::get_object_assert 'pod valid-pod' "{{.$id_field}}" 'valid-pod' </ins> ### Dump current valid-pod POD output_pod=$(kubectl get pod valid-pod -o yaml --output-version=v1beta1 "${kube_flags[@]}") ### Delete POD valid-pod by id # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete pod valid-pod "${kube_flags[@]}" # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create POD valid-pod from dumped YAML # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command echo "${output_pod}" | kubectl create -f - "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Delete POD valid-pod from JSON # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete -f examples/limitrange/valid-pod.json "${kube_flags[@]}" # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create POD redis-master from JSON # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Delete POD valid-pod with label # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert 'pods -l name=valid-pod' '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert 'pods -l name=valid-pod' "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete pods -l name=valid-pod "${kube_flags[@]}" # Post-condition: no POD is running <del> kube::test::get_object_assert 'pods -l name=valid-pod' '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert 'pods -l name=valid-pod' "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create POD valid-pod from JSON # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Delete PODs with no parameter mustn't kill everything # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command ! kubectl delete pods "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Delete PODs with --all and a label selector is not permitted # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command ! kubectl delete --all pods -l name=valid-pod "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Delete all PODs # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete --all pods "${kube_flags[@]}" # --all remove all the pods # Post-condition: no POD is running <del> kube::test::get_object_assert 'pods -l name=valid-pod' '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert 'pods -l name=valid-pod' "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create two PODs # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" kubectl create -f examples/redis/redis-proxy.yaml "${kube_flags[@]}" # Post-condition: valid-pod and redis-proxy PODs are running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'redis-proxy:valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'redis-proxy:valid-pod:' </ins> ### Delete multiple PODs at once # Pre-condition: valid-pod and redis-proxy PODs are running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'redis-proxy:valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'redis-proxy:valid-pod:' </ins> # Command kubectl delete pods valid-pod redis-proxy "${kube_flags[@]}" # delete multiple pods at once # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create two PODs # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" kubectl create -f examples/redis/redis-proxy.yaml "${kube_flags[@]}" # Post-condition: valid-pod and redis-proxy PODs are running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'redis-proxy:valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'redis-proxy:valid-pod:' </ins> ### Stop multiple PODs at once # Pre-condition: valid-pod and redis-proxy PODs are running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'redis-proxy:valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'redis-proxy:valid-pod:' </ins> # Command kubectl stop pods valid-pod redis-proxy "${kube_flags[@]}" # stop multiple pods at once # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create valid-pod POD # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Label the valid-pod POD # Pre-condition: valid-pod is not labelled <del> kube::test::get_object_assert 'pod valid-pod' '{{range.labels}}{{.}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{range.$labels_field}}{{.}}:{{end}}" 'valid-pod:' </ins> # Command kubectl label pods valid-pod new-name=new-valid-pod "${kube_flags[@]}" # Post-conditon: valid-pod is labelled <del> kube::test::get_object_assert 'pod valid-pod' '{{range.labels}}{{.}}:{{end}}' 'valid-pod:new-valid-pod:' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{range.$labels_field}}{{.}}:{{end}}" 'valid-pod:new-valid-pod:' </ins> ### Delete POD by label # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete pods -lnew-name=new-valid-pod "${kube_flags[@]}" # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ### Create valid-pod POD # Pre-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> # Command kubectl create -f examples/limitrange/valid-pod.json "${kube_flags[@]}" # Post-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> ### Overwriting an existing label is not permitted # Pre-condition: name is valid-pod <del> kube::test::get_object_assert 'pod valid-pod' '{{.labels.name}}' 'valid-pod' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{.${labels_field}.name}}" 'valid-pod' </ins> # Command ! kubectl label pods valid-pod name=valid-pod-super-sayan "${kube_flags[@]}" # Post-condition: name is still valid-pod <del> kube::test::get_object_assert 'pod valid-pod' '{{.labels.name}}' 'valid-pod' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{.${labels_field}.name}}" 'valid-pod' </ins> ### --overwrite must be used to overwrite existing label # Pre-condition: name is valid-pod <del> kube::test::get_object_assert 'pod valid-pod' '{{.labels.name}}' 'valid-pod' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{.${labels_field}.name}}" 'valid-pod' </ins> # Command kubectl label --overwrite pods valid-pod name=valid-pod-super-sayan "${kube_flags[@]}" # Post-condition: name is valid-pod-super-sayan <del> kube::test::get_object_assert 'pod valid-pod' '{{.labels.name}}' 'valid-pod-super-sayan' </del> <ins> kube::test::get_object_assert 'pod valid-pod' "{{.${labels_field}.name}}" 'valid-pod-super-sayan' </ins> ### Delete POD by label # Pre-condition: valid-pod POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' 'valid-pod:' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" 'valid-pod:' </ins> # Command kubectl delete pods -lname=valid-pod-super-sayan "${kube_flags[@]}" # Post-condition: no POD is running <del> kube::test::get_object_assert pods '{{range.items}}{{.id}}:{{end}}' '' </del> <ins> kube::test::get_object_assert pods "{{range.items}}{{.$id_field}}:{{end}}" '' </ins> ##############

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: examples/storage/cassandra/image/run.sh:	cluster_name  examples/storage/vitess/env.sh:    node_ip=$(get_node_ip) hack/jenkins/e2e-runner.sh:    local -r image_project="$1" <ins> hack/local-up-cluster.sh:        advertise_address="--advertise_address=${API_HOST}" </ins> hack/local-up-cluster.sh:      runtime_config="--runtime-config=${RUNTIME_CONFIG}" <ins> hack/local-up-cluster.sh:    advertise_address="" </ins> hack/local-up-cluster.sh:    runtime_config="" hack/test-update-storage-objects.sh:  local storage_media_type=${2:-""} hack/test-update-storage-objects.sh:  local storage_versions=${1:-""}

seems to have broken the kubernetes service when using : With this patch: Without it: Tested with v 1.3.2.
Looks like we don't like explicitly don't want to allow 127.0.0.1 loopback address here -


Question: func testPodFailSubpath(f *framework.Framework, pod *v1.Pod) { <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred(), "while creating pod") </del> <ins> var podClient *framework.PodClient podClient = f.PodClient() pod = podClient.Create(pod) </ins> defer func() { framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </del> <ins> err := framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </ins> Expect(err).To(HaveOccurred(), "while waiting for pod to be running") }

Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test


Question: "strings" "testing" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client/clientcmd" clientcmdapi "github.com/GoogleCloudPlatform/kubernetes/pkg/client/clientcmd/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/util"

E.g., it calls Convert() for every byte of a slice of uint8. E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting []v1.NamedExtension E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting []v1.NamedCluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting v1.Cluster E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting bool E0330 22:15:23. 5352 ] TIM: converting string E0330 22:15:23. 5352 ] TIM: converting []uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 E0330 22:15:23.

5352 ] TIM: converting uint8 E0330 22:15:23. 5352 ] TIM: converting uint8 ... thousands of these.
There's an easy fix for this :/


Question: t.Errorf("Expect latency to be greater than 0, got: %v", m.Histogram.SampleSum) } } <ins> func collectAndComparePermitWaitDuration(t *testing.T, wantRes string) { m := collectHistogramMetric(metrics.PermitWaitDuration) if wantRes == "" { if m != nil { t.Errorf("PermitWaitDuration shouldn't be recorded but got %+v", m) } return } if wantRes != "" { if len(m.Label) != 1 { t.Fatalf("Unexpected number of label pairs, got: %v, want: 1", len(m.Label)) } if *m.Label[0].Value != wantRes { t.Errorf("Unexpected result label, got: %q, want %q", *m.Label[0].Value, wantRes) } if *m.Histogram.SampleCount != 1 { t.Errorf("Expect 1 sample, got: %v", m.Histogram.SampleCount) } if *m.Histogram.SampleSum <= 0 { t.Errorf("Expect latency to be greater than 0, got: %v", m.Histogram.SampleSum) } } } func collectHistogramMetric(metric prometheus.Collector) *dto.Metric { ch := make(chan prometheus.Metric, 1) metric.Collect(ch) select { case got := <-ch: m := &dto.Metric{} got.Write(m) return m default: return nil } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : metric name Description -- -- permitwaitduration_seconds Duration of permit extension point finishes waiting. Why is this needed: In permit extension point, if the status is "Wait", we block until the pod is scheduled or timeout: Blocking is an important behavior that we should monitor. /sig scheduling
/assign


Question: } return nil, fmt.Errorf("bad pod %s", name) } <del> resp, err := http.Get(fw.testHTTPServer.URL + "/podInfo?podID=goodpod") </del> <ins> resp, err := http.Get(fw.testHTTPServer.URL + "/podInfo?podID=goodpod&podNamespace=default") </ins> if err != nil { t.Errorf("Got error GETing: %v", err) }

The podInfo call has no way to accept namespace, which means the apiserver can't fetch pod info. Probably need to make namespace a required path parameter.
this is probably you :)
- is this something you see when working off ??
- ok so this is pod_cache, will review
If you don't time, I can pick up this one.
I got it.
Per discussion with , will deliver a pull after is merged.


Question: return code, nil } <del> // DialFromNode executes a tcp or udp request based on protocol via kubectl exec </del> <ins> // DialFromNode executes a tcp/udp curl/nc request based on protocol via kubectl exec </ins> // in a test container running with host networking. <del> // - minTries is the minimum number of curl attempts required before declaring //   success. Set to 0 if you'd like to return as soon as all endpoints respond //   at least once. // - maxTries is the maximum number of curl attempts. If this many attempts pass //   and we don't see all expected endpoints, the test fails. // maxTries == minTries will confirm that we see the expected endpoints and no // more for maxTries. Use this if you want to eg: fail a readiness check on a // pod and confirm it doesn't show up as an endpoint. func (config *NetworkingTestConfig) DialFromNode(protocol, targetIP string, targetPort, maxTries, minTries int, expectedEps sets.String) { </del> <ins> // - minTries is the minimum number of curl/nc attempts required before declaring //   success. If 0, then we return as soon as all endpoints succeed. // - There is no logical change to test results if faillures happen AFTER endpoints have succeeded, //   hence over-padding minTries will NOT reverse a successful result and is thus not very useful yet //   (See the TODO about checking probability, which isnt implemented yet). // - maxTries is the maximum number of curl/echo attempts before an error is returned.  The //   smaller this number is, the less 'slack' there is for declaring success. // - if maxTries < expectedEps, this test is guaranteed to return an error, because all endpoints wont be hit. // - maxTries == minTries will return as soon as all endpoints succeed (or fail once maxTries is reached without //   success on all endpoints). //   In general its prudent to have a high enough level of minTries to guarantee that all pods get a fair chance at receiving traffic. func (config *NetworkingTestConfig) DialFromNode(protocol, targetIP string, targetPort, maxTries, minTries int, expectedEps sets.String) error { </ins> var cmd string if protocol == "udp" { cmd = fmt.Sprintf("echo hostName | nc -w 1 -u %s %d", targetIP, targetPort)

What happened: final passing output: What you expected to happen: Probably if this test passes, it should be showing several real endpoints. I don't know exactly what it does, but im assuming that the output could be clarified a little. How to reproduce it (as minimally and precisely as possible): run and look at the logs. Anything else we need to know?: Environment: Kubernetes version (use ): 1.20 (sonobuoy)
/sig-network
/assign
/triage accepted
^ an interesting case here was that was but there was no SIG (typo in the command that did). should the label be only if there is label on the ticket? /sig network
the PR that adds triage/accepeted automation () was merged 3 hours ago, but it'll "actually" get rolled out in the next prow bump. Mostly when merges. Thanks for flagging though, appreciate it! :)
sorry, missed the actual question! :woman_facepalming: (need moar coffee!! :coffee: ) I don't think we should require a label for adding the label. In case someone comments with the following, we'd still want it to be valid.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The above test cleans up the By directives and so on, After reading through the test more its clear that every test sets up a scaffold measures a property of that scaffold sometimes deletes something that is recovered in that scaffold (i.e. a pod that is behind a UDP service) So, the thing that was confusing I guess to me is that we validate empty conditions (i.e. no pods available) once we delete a nodePort svc. Meanwhile also noticed that the original work to clean up error reporting needed to cascade over to these tests as well.Have just finished all that, and also updated comments to reflect the Todo around the statistical computation (we have the ability to do extra polling via but we don't actually have any test that leverages this.As a follow on I guess it would be nice if we did have a test that confirmed that pods were load balanced evenly, esp with the new proliferation of different proxy technologies , I.e. a (or something similar


Question: if stderr != nil { command.Stderr = stderr } <del> return command.Run() </del> <ins> return newRktExitError(command.Run()) </ins> } // PortForward executes socat in the pod's network namespace and copies

With rkt as the container runtime, the e2e test fails with I dove into it a bit further and it looks like a real issue. With additional logging it appears that the probe does fail (exec happens and returns incorrectly as expected), but when calls it never gets that the given ID exists. The pod definition I've been using is: It restarts very slowly with rkt (on the order of an hour I believe, see sleep 600), but restarts very frequently with docker. cc


Question: } } <ins> func TestStatusNormalizeTimeStamp(t *testing.T) { pod := v1.Pod{ Spec: v1.PodSpec{}, } now := metav1.Now() podStatus := v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: now}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: now, FinishedAt: now}}}, }, InitContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: now}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: now, FinishedAt: now}}}, }, EphemeralContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: now}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: now, FinishedAt: now}}}, }, } expectedTime := now.DeepCopy().Rfc3339Copy() expectedPodStatus := v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: expectedTime}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: expectedTime, FinishedAt: expectedTime}}}, }, InitContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: expectedTime}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: expectedTime, FinishedAt: expectedTime}}}, }, EphemeralContainerStatuses: []v1.ContainerStatus{ {State: v1.ContainerState{Running: &v1.ContainerStateRunning{StartedAt: expectedTime}}}, {State: v1.ContainerState{Terminated: &v1.ContainerStateTerminated{StartedAt: expectedTime, FinishedAt: expectedTime}}}, }, } normalizedStatus := normalizeStatus(&pod, &podStatus) if !isPodStatusByKubeletEqual(&expectedPodStatus, normalizedStatus) { t.Fatalf("The timestamp is not correctly converted to RFC3339 format.") } } </ins> func TestStaticPod(t *testing.T) { staticPod := getTestPod() staticPod.Annotations = map[string]string{kubetypes.ConfigSourceAnnotationKey: "file"}

is not normalized in in the status manager: This does not seem to cause any user facing problem because is sorted before passed to the status manager and timestamps are normalized at marshaling to create a patch: However, an unexpected behavior is caused internally because timestamps are not normalized. If the log verbosity is 3 or larger, the following messages are logged periodically in kubelet when an ephemeral container exists: This message is emitted here: After returns , is called. This does not look so harmful because gets eventually: API is called once unnecessarily in , though: It might be better to normalize ephemeral container statuses, at least timestamps, in the status manager to avoid this unexpected behavior. Set the log verbosity of kubelet to 3 or larger. Then, create an ephemeral container like . No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
This is a bug. Can you submit a PR to fix it? If not, I'd be happy to work on it
If you are interested, please feel free to assign yourself.
thx
/assign
/triage accepted /priority important-longterm


Question: // Filters the nodes to find the ones that fit based on the given predicate functions // Each node is passed through the predicate functions to determine if it is a fit <del> func (g *genericScheduler) findNodesThatFit(pluginContext *framework.PluginContext, pod *v1.Pod, nodes []*v1.Node) ([]*v1.Node, FailedPredicateMap, error) { </del> <ins> func (g *genericScheduler) findNodesThatFit(pluginContext *framework.PluginContext, pod *v1.Pod, nodeLister algorithm.NodeLister) ([]*v1.Node, FailedPredicateMap, error) { </ins> var filtered []*v1.Node failedPredicateMap := FailedPredicateMap{} if len(g.predicates) == 0 { <del> filtered = nodes </del> <ins> filtered = nodeLister.ListNodes() </ins> } else { allNodes := int32(g.cache.NodeTree().NumNodes()) numNodesToFind := g.numFeasibleNodesToFind(allNodes)

In : It seems that nodes are only used in the case there are no predicates. This call can be moved inside . Moving it there saves us from creating the node slice unless it is necessary. Checking if there are no nodes (line 202 below) can be done using . /kind cleanup /sig scheduling
I can work on this


Question: "bytes" "fmt" "net/http" <ins> "reflect" </ins> "strings" "sync" "sync/atomic"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: }(plugins[i]) } <ins> w := newWatcherWithHandler(t, hdlr, false /* testDeprecatedDir */) defer func() { require.NoError(t, w.Stop()) }() </ins> c := make(chan struct{}) go func() { defer close(c)

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: } func TestParseACRLoginServerFromImage(t *testing.T) { <ins> configStr := ` { "aadClientId": "foo", "aadClientSecret": "bar" }` result := []containerregistry.Registry{ { Name: to.StringPtr("foo"), RegistryProperties: &containerregistry.RegistryProperties{ LoginServer: to.StringPtr("*.azurecr.io"), }, }, } fakeClient := &fakeClient{ results: result, } provider := &acrProvider{ registryClient: fakeClient, } provider.loadConfig(bytes.NewBufferString(configStr)) provider.environment = &azure.Environment{ ContainerRegistryDNSSuffix: ".azurecr.my.cloud", } </ins> tests := []struct { image    string expected string

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: case !shouldContinueRunning && exists: // If daemon pod isn't supposed to run on node, but it is, delete all daemon pods on node. for _, pod := range daemonPods { <ins> if pod.DeletionTimestamp != nil { continue } </ins> podsToDelete = append(podsToDelete, pod.Name) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
/sig apps
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, RestartPolicy: v1.RestartPolicyNever, <ins> NodeSelector:  nodeSelector, </ins> }, } By(fmt.Sprintf("Creating pod %s", pod.Name))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Test pod is not created successfully, likely due to it being scheduled in a different zone from where the PD is provisioned. <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig storage /assign /priority critical-urgent


Question: return nil } <del> func (r RealPodControl) DeletePod(namespace, podID string) error { return r.KubeClient.Pods(namespace).Delete(podID, nil) </del> <ins> func (r RealPodControl) DeletePod(namespace string, podID string, object runtime.Object) error { meta, err := api.ObjectMetaFor(object) if err != nil { return fmt.Errorf("object does not have ObjectMeta, %v", err) } if err := r.KubeClient.Pods(namespace).Delete(podID, nil); err != nil { r.Recorder.Eventf(object, api.EventTypeWarning, "FailedDelete", "Error deleting: %v", err) return fmt.Errorf("unable to delete pods: %v", err) } else { glog.V(4).Infof("Controller %v deleted pod %v", meta.Name, podID) r.Recorder.Eventf(object, api.EventTypeNormal, "SuccessfulDelete", "Deleted pod: %v", podID) } return nil </ins> } type FakePodControl struct {

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc


Question: return false } <del> // Allocates from reusableDevices list first. if allocateRemainingFrom(reusableDevices) { return allocated, nil } </del> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.NewString() } <ins> // Allocates from reusableDevices list first. if allocateRemainingFrom(reusableDevices) { return allocated, nil } </ins> // Gets Devices in use. devicesInUse := m.allocatedDevices[resource] // Gets Available devices.

kubelet panic with nil map when apiserver restart and kubelet is allocating scaler resources. ! kubelet should not panic. when kubelet is allocating scaler resource with pod starting, restart the api servers; No response <detailsv1.23.6 </details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
Can this be reproduced? Could you share some more information to reproduce it? After kubelet restart, does it happen again?
/triage needs-information can you please give more information on the set up and how it got to this state. Also if you can include , it will help diagnotics
/assign
fixes the crash by allocating prior to calling allocateRemainingFrom.
/triage accepted /priority important-longterm
/remove-triage needs-information
Are master and minion in the same node? Normally api-server desn't influnce kubelet. ! !


Question: generate-etcd-cert "${KUBE_TEMP}/cfssl" "${hostClient}" "client" "etcd-apiserver-client" pushd "${KUBE_TEMP}/cfssl" <del> ETCD_APISERVER_CA_KEY_BASE64=$(base64 "ca-key.pem" | tr -d 'rn') </del> <ins> ETCD_APISERVER_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "ca-key.pem" | tr -d 'rn') </ins> ETCD_APISERVER_CA_CERT_BASE64=$(gzip -c "ca.pem" | base64 | tr -d 'rn') <del> ETCD_APISERVER_SERVER_KEY_BASE64=$(base64 "etcd-apiserver-server-key.pem" | tr -d 'rn') </del> <ins> ETCD_APISERVER_SERVER_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "etcd-apiserver-server-key.pem" | tr -d 'rn') </ins> ETCD_APISERVER_SERVER_CERT_BASE64=$(gzip -c "etcd-apiserver-server.pem" | base64 | tr -d 'rn') <del> ETCD_APISERVER_CLIENT_KEY_BASE64=$(base64 "etcd-apiserver-client-key.pem" | tr -d 'rn') </del> <ins> ETCD_APISERVER_CLIENT_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "etcd-apiserver-client-key.pem" | tr -d 'rn') </ins> ETCD_APISERVER_CLIENT_CERT_BASE64=$(gzip -c "etcd-apiserver-client.pem" | base64 | tr -d 'rn') popd }

under kubernetes project and running get error: ! I am running on darwin arm64 running get this: if running on linux:
/area test
/sig cli
/sig testing /area test-infra /remove-sig cli /remove-area test
FWIW You pretty much have to install gnu coreutils to develop here, many scripts will actually refuse to run if the gnu version of utils are not available. That said, we should fix this.
/triage accepted


Question: dryRunStrategy   cmdutil.DryRunStrategy dryRunVerifier   *resource.DryRunVerifier builder          *resource.Builder <del> generatorFunc    func(string, *meta.RESTMapping) (generate.StructuredGenerator, error) </del> <ins> generatorFunc    func(name, refName string, mapping *meta.RESTMapping) (generate.StructuredGenerator, error) </ins> fieldManager     string HPAClient         autoscalingv1client.HorizontalPodAutoscalersGetter

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: the hpa name is always info.Namenot --name What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig cli


Question: // Validate federation apiserver, does not rely on underlying clusters or federation ingress controller. Describe("Ingress objects", func() { AfterEach(func() { nsName := f.FederationNamespace.Name // Delete registered ingresses. ingressList, err := f.FederationClientset_1_4.Extensions().Ingresses(nsName).List(api.ListOptions{})

The addition of the federated namespace adminssion controllers broke this test. It's an easy fix. e2e test needs to create the required namespace before trying to create things in it. FYI. has volunteered to fix this. https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e

suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: []

Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
is fejta@ the right owner?
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated

ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated

Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in

underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e


Question: } if exist := waitForPathToExist(&devicePath, multipathDeviceTimeout, iscsiTransport); !exist { <del> klog.Errorf("Could not attach disk: Timeout after 10s") </del> <ins> klog.Errorf("Could not attach disk: Timeout after %ds", multipathDeviceTimeout) </ins> // update last error <del> lastErr = fmt.Errorf("Could not attach disk: Timeout after 10s") </del> <ins> lastErr = fmt.Errorf("Could not attach disk: Timeout after %ds", multipathDeviceTimeout) </ins> continue } else { devicePaths[tp] = devicePath

What happened: Minor issue with the current source code in : The error message "Could not attach disk: Timeout after 10s" includes the verbatim string constant "10" (appears two times in adjacent rows), but the call to in the preceding if statement refers to the Go constant (currently set to 10) instead. The strings for the error messages should refer to (i.e. print) the same Go constant to guarantee that they always stay in sync. Kubernetes version: current master branch on GitHub /sig storage
/assign


Question: e2epod.WaitForPodsRunningReady(ctx, f.ClientSet, f.Namespace.Name, 1, 0, timeout) ginkgo.By("Getting container stats for pod") <del> nodeStats, err := e2ekubelet.GetStatsSummary(ctx, f.ClientSet, targetNode.Name) framework.ExpectNoError(err, "Error getting node stats") </del> statsChecked := false <del> for _, podStats := range nodeStats.Pods { if podStats.PodRef.Namespace != f.Namespace.Name { continue } // check various pod stats if *podStats.CPU.UsageCoreNanoSeconds <= 0 { framework.Failf("Pod %s/%s stats report cpu usage equal to %v but should be greater than 0", podStats.PodRef.Namespace, podStats.PodRef.Name, *podStats.CPU.UsageCoreNanoSeconds) } if *podStats.Memory.WorkingSetBytes <= 0 { framework.Failf("Pod %s/%s stats report memory usage equal to %v but should be greater than 0", podStats.PodRef.Namespace, podStats.PodRef.Name, *podStats.CPU.UsageCoreNanoSeconds) } for _, containerStats := range podStats.Containers { statsChecked = true // check various container stats if *containerStats.CPU.UsageCoreNanoSeconds <= 0 { framework.Failf("Container %s stats report cpu usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.CPU.UsageCoreNanoSeconds) } if *containerStats.Memory.WorkingSetBytes <= 0 { framework.Failf("Container %s stats report memory usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.Memory.WorkingSetBytes) } if *containerStats.Logs.UsedBytes <= 0 { framework.Failf("Container %s stats log usage equal to %v but should be greater than 0", containerStats.Name, *containerStats.Logs.UsedBytes) } } } </del> <ins> wait.Poll(2*time.Second, timeout, func() (bool, error) { return ensurePodsStatsOnNode(ctx, f.ClientSet, f.Namespace.Name, targetNode.Name, &statsChecked) }) </ins> if !statsChecked { framework.Failf("Failed to get stats for pod %s/%s", f.Namespace.Name, podName)

ci-kubernetes-e2e-capz-master-containerd-windows Kubernetes e2e suite: [It] [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers container stats validation feb 8th 2023 reading of the log is showing reporting zero It looks like the pod had started up just as it check the failure on the test. Adding a retry or a longer wait might make the test more reliable /sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: {% if grains['os_family'] != 'RedHat' %} - file: /etc/init.d/kubelet {% endif %} <ins> {% if grains.cloud is defined %} {% if grains.cloud == 'gce' %} </ins> - file: /var/lib/kubelet/kubernetes_auth <ins> {% endif %} {% endif %} </ins>

The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver.

I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.


Question: nodeInfo: schedulercache.NewNodeInfo( newResourcePod(schedulercache.Resource{MilliCPU: 10, Memory: 10, StorageOverlay: 20})), fits: false, <del> test: "due to init container scratch disk", </del> <ins> test: "due to container scratch disk", </ins> reasons: []algorithm.PredicateFailureReason{ NewInsufficientResourceError(v1.ResourceCPU, 1, 10, 10), NewInsufficientResourceError(v1.ResourceStorageScratch, 1, 20, 20),

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local isolation Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Bug Kubernetes version (use ): kubernetes HEAD Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: For local storage isolation, overlay resource predicates for pod with only overlay or scratch storage request doesn't work. What you expected to happen: If a pod requests only overlay or scratch storage, then local storage isolation feature should limit the pods. How to reproduce it (as minimally and precisely as possible): Run local cluster from master branch, and create following pods (against a machine with less overlay fs than 1024Gi). Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
CC
Fix is ready:


Question: framework.ExpectNoError(err, fmt.Sprintf("error while waiting for service:%s err: %v", serviceName, err)) hostExec := utils.NewHostExec(f) defer hostExec.Cleanup() <del> node, err := e2enode.GetRandomReadySchedulableNode(cs) framework.ExpectNoError(err) </del> <ins> node := &nodes.Items[0] </ins> cmd := "iptables-save" if framework.TestContext.ClusterIsIPv6() { cmd = "ip6tables-save"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- - At least 08/08 sequentially performs operations that take ~20s on average. Example: For a 5000 nodes test this causes a major timeout. excludes tests with tag which SCTP had until was merged. This is why there is a shift in red cells in the testgrid after 08/15. We will add a pattern excluding until this issue is resolved. /assign /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign


Question: // Apply implements Manager. func (f *managedFieldsUpdater) Apply(liveObj, appliedObj runtime.Object, managed Managed, fieldManager string, force bool) (runtime.Object, Managed, error) { <del> formerManaged := managed.Fields().Copy() </del> object, managed, err := f.fieldManager.Apply(liveObj, appliedObj, managed, fieldManager, force) if err != nil { return object, managed, err } <del> if object != nil || !managed.Fields().Equals(formerManaged) { </del> <ins> if object != nil { </ins> managed.Times()[fieldManager] = &metav1.Time{Time: time.Now().UTC()} <del> } if object == nil { </del> <ins> } else { </ins> object = liveObj.DeepCopyObject() internal.RemoveObjectManagedFields(object) }

I created an object with , and later I updated the same object with another . After that I looked at the object's managedFields, and saw nothing about the update. Here is what I got after the second apply: I expected managedFields to get updated by the second . No response <details</details<detailsnone </details<details</details<detailskubeadm </details<detailsdocker </details<details</details
/cc
I also saw the timestamp not updated when I updated using client-go and typed client stubs, Apply method, patch type .
/assign /triage accepted
/cc
The is not updated because you are not changing the structure of the object (adding or removing fields). According to , the time is updated if and only if the fieldSet is changed. It does not take into account the value of fields. One option would be to not compare the fieldSets but directly compare the liveObject & newObject for differences and update the field accordingly. Is it the desired behavior to update the time of when only the value is updated?
Yes! Look at the first paragraph of .
is that documentation correct? Is this what we intended?
I don't see any mention to the timestamp in that first paragraph. We've clarified recently the exact semantic of the timestamp, as defined here: I think you've identified the wrong code if you want to go ahead and fix it, go ahead, I'll happily review!


Question: } var username string <del> if err := c.unmarshalClaim(a.jwtAuthenticator.ClaimMappings.Username.Claim, &username); err != nil { return nil, false, fmt.Errorf("oidc: parse username claims %q: %v", a.jwtAuthenticator.ClaimMappings.Username.Claim, err) </del> <ins> usernameClaim := a.jwtAuthenticator.ClaimMappings.Username.Claim if err := c.unmarshalClaim(usernameClaim, &username); err != nil { return nil, false, fmt.Errorf("oidc: parse username claims %q: %v", usernameClaim, err) </ins> } <del> if a.jwtAuthenticator.ClaimMappings.Username.Claim == "email" { </del> <ins> if usernameClaim == "email" { </ins> // If the email_verified claim is present, ensure the email is valid. // https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims if hasEmailVerified := c.hasClaim("email_verified"); hasEmailVerified {

Follow up PR: would be good to use some local vars to make these statements shorter. Originally posted by in
/sig auth /triage accepted /kind cleanup /assign


Question: } } <del> if *serverVersion { </del> <ins> if *serverVersion != verflag.VersionFalse { </ins> got, err := kubeClient.ServerVersion() if err != nil { fmt.Printf("Couldn't read version from server: %vn", err) os.Exit(1) } <del> fmt.Printf("Server Version: %#vn", got) os.Exit(0) </del> <ins> if *serverVersion == verflag.VersionRaw { fmt.Printf("%#vn", *got) os.Exit(0) } else { fmt.Printf("Server: Kubernetes %sn", got) os.Exit(0) } </ins> } if *preventSkew {

It's hard to tell if they match right now.
who has been changing this stuff. :)
Ah yes... I was thinking of adding a flag to print the %r of the version.Info... Do you think that would be an appropriate flag name? I'll start working on the PR since it's probably time to refactor the code to take the other bits from git I have been pushing through ldflags...
I'd change the current behavior rather than add a new flag. I don't really care which way it ends up looking as long as both flags print the same format :)
Fixed by