Question: } } <del> // If all metrics are invalid return error and set condition on hpa based on first invalid metric. if invalidMetricsCount >= len(metricSpecs) { </del> <ins> // If all metrics are invalid or some are invalid and we would scale down, // return an error and set the condition of the hpa based on the first invalid metric. // Otherwise set the condition as scaling active as we're going to scale if invalidMetricsCount >= len(metricSpecs) || (invalidMetricsCount > 0 && replicas < specReplicas) { </ins> setCondition(hpa, invalidMetricCondition.Type, invalidMetricCondition.Status, invalidMetricCondition.Reason, invalidMetricCondition.Message) return 0, "", statuses, time.Time{}, fmt.Errorf("invalid metrics (%v invalid out of %v), first error is: %v", invalidMetricsCount, len(metricSpecs), invalidMetricError) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If you have 2 or more metrics defined on an HPA and one metric is suggesting to scale down and the other metric is unavailable, the HPA will scale down despite missing metrics. Example: Imagine the following Horizontal Pod Autoscaler resource: And the deployment is currently scaled to replicas. If the metric is suggesting to scale to 1 AND the metric is unavailable e.g. HPA can't fetch the metrics. Then it will happily scale down to despite not knowing what the currently unavailable metric would suggest. It's clearly stated in the docs the scale down should not happen if one or more metrics are unavailable: Until Kubernetes v1.15.x the behavior was as expected. However with the . The logic was removing the special handling of stopping scale down in case of unavailable metrics. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /bug /sig autoscaling
/triage accepted


Question: type Context struct { // LocationOfOrigin indicates where this object came from.  It is used for round tripping config post-merge, but never serialized. // +k8s:conversion-gen=false <del> LocationOfOrigin string </del> <ins> LocationOfOrigin string `json:"-"` </ins> // Cluster is the name of the cluster for this context Cluster string `json:"cluster"` // AuthInfo is the name of the authInfo for this context

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> FROM scratch LABEL maintainer "Shyam JVS <shyamjvs@google.com>" </del> <ins> FROM gcr.io/distroless/static:latest </ins> COPY etcd-version-monitor /etcd-version-monitor

EDIT (from - 5/8/20): [x] () New go-runner image for distroless scenarios [x] () Promote the new image - [x] () Bump to [x] () Promote image [x] () Switch core master base images from debian to distroless [x] () Flexvolume master driver is blocking controller-manager from moving to distroless - [x] () Base image (debian-base, debian-iptables) updates & building on K8s Infra [x] () 'pause' base image updates & building on K8s Infra [x] () Remove image [ ] () Remove image Forked from (think of this as V2). is not quite sufficient as a base image for static go binaries. Common dependencies include ca-certificates, , hosts, and nsswtich (see ). See below for a more complete list. We need to decide if we want a scratch+config base image (either found or created by us), or a more complete distro (e.g. debian-base, at the cost of more frequent patching). has also been proposed, or could provide inspiration for a static base. Blocked by: (for the scratch case) , for logs redirection. /kind bug /cc Appendix Here is a list of files referenced from the go source (obviously some of these are not applicable):
/sig cluster-lifecycle
not sure if sig cluster lifecycle is the best owner for this. adding sig release and sig architecture. /sig release /sig architecture
I just hit during a deployment of 1.12, this issue: Please add a sane to all the images or convince the golang folks to fix their defaults.
/assign This is not sig-cluster-lifecycle typically.
is not enough?
before there was , though that didn't make it into 1.12. cc
Yeah, I an as a small stopgap in , but I'd like to see a standard base image as well which should simplify the build and patching a bit. We could backport or to 1.12, I'm not sure what the best choice is. is smaller but is the preferable change going forward.

Note that go's defaults actually match the documentation for not having an : (), it's just that they don't match the default most distros seem to ship of which is probably what most of us want ...
Another option is to just use , although it does include a few c libraries that we don't need.
would distroless/base have what we need for EG the iptables (kubeproxy) variant?
I don't think so - it doesn't have a package manager so iptables would still need to be manually (and managed).
It looks like distroless is planning to put out an image without the c libraries, specifically for this use case. I think we should just use that as soon as it's ready:
Update: is now available. I suggest we work towards this end-state: All official Kubernetes images & addons are based on 1 of 2 images: wherever possible, or a full featured distro for the cases where it's absolutely required. For the latter, our current recommendation is , but that is subject to change. For these cases, the dependencies should be well-documented, and we should eventually work towards building from source, and basing on a distroless image. Action Items: : DONE generate a list of images needing updates for each item below document decision in a more "official" place (community repo) all images on . There are some known-issues with running go static binaries on scratch, and this move should fix those (e.g. ). system images using shell-redirection for logs on distroless/static, by leveraging option to eliminate shell usage. (Requires v1.13+) away at the remaining images with miscellaneous shell dependencies. In some cases, the shell usage should be rewritten as a feature of the binary, in others it's a harder dependency that should be documented and migrated to debian-base (see above). We should have separate tracking issues for fixing these on a case-by-case basis. I propose we try to tackle 1-3 in Kubernetes 1.14. /milestone v1.14 /help
cc (please see 1i from Tim above)
I would like to start digging into this.
SGTM, one thing worth noting is that a few related projects have used (one of mine used to), wish we had a good way to signal intent regarding the future of that image.

it seems like it might be a bit late to tackle this for 1.14 (cc though it might be worth starting on a KEP to outline this in any case?
what would you want to a see in a KEP about this? Just the justification, and an analysis of the risks in moving to the new base image?
/assign
GitHub didn't allow me to assign the following users: yuwenma. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as does not contain any platform specific code and just contains: So we are safe using this instead of scratch wherver needed.
We're well into code freeze for 1.14, is this necessary/release-blocking? Also, this seems to be a major enhancement without a KEP as mentioned above. Based on the work defined above that's appears to be remaining and there is no KEP associated with this do you mind if we push this to 1.15?
Definitely not release-blocking. /milestone v1.15 it sounds like there's demand for a KEP. Would you mind translating your design doc into a KEP? Follow for guidelines.
Will do that. Right now, I'm referring to this issue for the full context but I think KEP is a good idea.
/remove-sig architecture
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
Hi Marko, thanks for the check. Yes! This is planed. We have had the majority of the work done. Have three more PRs need to go in (under review). Please note, this is not a feature request but more of a bug-fix/internal-cleanup due to the security reasons. Does it have to strictly follow the code freeze policy?

Hello, I'm the Bug Triage Lead for 1.15. All issues and PRs are subject to the Code Freeze policy, this is to try and maintain as much stability as possible while preparing a new release. Last release there were comments about creating a KEP for this work, was that ever completed? If so, if your code is complete, and a documentation PR is open then this will likely land in this release. If a KEP has not been created this issue, and associated PRs, may still be incorporated in this release during the Code Thaw phase. For more information please refer to this doc:
Thanks for the reminder. yeah, we are following the policy and here's the We have the PRs ready and they depend on the latest klog (v0.3.2, created yesterday). I'm currently working on importing the new klog version to k/k ( the new klog release may require some test updates) and afterwords we can have the blocking PR merged. Due to our current situation, do you have any suggestions what can be the good release phase for us? should we go with the Code Thaw phase?
If you can get the Klog version updated and the PRs merged by EOD tomorrow (May 30th) you will be fine, otherwise shoot for Code Thaw. I recommend trying to pull in the appropriate Sig leadership to help review/expedite your PRs getting merged
Hello once again! I'd just like to check what's the current status of this issue and is it still planned for v1.15? Thanks!
's the status updates: We have rebased. And here's the .
It looks like Klog is still an issue, can we move this to 1.16?
Since I didn't receive a response I'm bumping this to 1.16 /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
Any update on this?

Should we move it to the next milestone?
/milestone v1.17 This is an ongoing issue with incremental improvements in each release.
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still on track for 1.17?
/milestone v1.18
Context: sig-architecture subgroup work to search for technical debt Category: Security Reason: Distroless containers decreased attack surface
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/milestone v1.19 /remove-lifecycle stale switching milestone to next cycle. please change back if need.
- have the klog issues that were blocking some of the control plane images been resolved? /cc
we haven't tracked it down yet. last debugging was here -
I've updated the PR description with some of the current work Dims and I are doing. /assign /remove-help /remove-priority important-longterm /priority important-soon
FYI
Hello, here from Bug Triage! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting June 25th and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
-- Yep, it's active for this release. The issue description and tracked items was updated 18 days ago and there are several merged PRs linking back to this.
I still see a couple open PR's. Due to reaching the code freeze deadline, I'm going to bump this to the next release. /milestone v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Heyho I wanted to check how it is going? As the issue is tagged for 1.20, is it still planned for this release?
This is roughly done, but I need to backfill docs. /milestone v1.21
Hi , I'm from the Bug Triage team, is still planned for 1.21? We start code freeze today, so I wonder if we should leave it for this release or move to the next
/milestone v1.22
Is there any update on this? Are the docs backfilled already?
Hey there ( ), Bug-Triage here , looks like this issue hasn't received any update since it has been assigned to . Given the fact that the code freeze for this release already started and the issue has been transitioned a couple of releases yet, I´d like to suggest clearing the milestone label for now. We will remove the label if this issue doesn't receive any significant update in 7d from now.
/milestone clear
I'm prototyping what distroless + additional necessary c binaries (like iptables) would look like in the KIND subproject here: It's not bad actually, especially given that distroless has images intended for C, C++ applications (adds libc6, adds libgcc1). I've been writing a generalizable approach to automatically copy binary + runtime dynamic library deps + their copyrights based on inspecting binaries with in a multi-stage build.
my concern with this approach is that it will essentially hide the packages & versions from vulnerability scanners, which I think mostly just look for package manager manifests. I wonder how hard it would be to keep a package manifest with the relevant packages in the image? Also, how does your approach compare to using bazel to build docker imagse?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Still needs a review of docs; will work on this during some intercycle downtime /remove-lifecycle rotten /lifecycle frozen /milestone v1.24
hi, I'm with the Bug triage team for 1.24 - we noticed that this issue has not been updated for a while. I wonder if there is any work planned or expected that would go into 1.24? Thanks!
I don't believe there are any further changes in 1.24. Sorry I missed your reply I don't think that will be hard at all. Distroless has pretty much the same proposition when using anything beyond "base", e.g. the "static" image has glibc from debian packaging. If we can settle on what information we wish to preserve (or generate!) we should be able to do this, e.g. currently we preserve package copyright info. And presumably distroless either doesn't handle this and leaves us in the same boat with other images, or we can copy what they're doing. I'd have to dig more to see, I think this is which is in the distroless base image, we should be able to preserve those in the same way we preserve copyright. Well, currently I'm using a multi-stage dockerfile but the overall approach can be implemented many ways. Multi-stage dockerfiles can be pinned to pull source images by digest and are relatively hermetic since they're inherently a sandboxed filesystem. If we have any concerns with this, we could also pull the package contents directly and add them to an image.

The bazel rules_docker package largely wraps Bazel is also slightly a moot point for kubernetes following
nudged me about this, digging around shows that distroless images persist package versions by putting the package control file in , which is close to normal dpkg (one big file with a blank line between packages, and some additional info beyond the control file) Source: Scanners already need to implement support for this to handle our other distroless images. (see e.g. ) We can very easily grab the package status () and stage it (into like bazel) along with the copyright and libraries we need for each binary, you can see this to my prototype here: (first commit in )
I think if we wanted to continue with this approach for the iptables image it would look like: 1) rewrite the iptables / iptables-nft select hack script to be a static go binary instead of a shell script 2) convert the image build to look like kubernetes-sigs/kind images/haproxy, but instead of haproxy do the iptables binaries and stage them and their libraries, add the version select tool + symlinks Then we'd have an image with just iptables binaries, their library dependencies, tzdata etc. from distroless base, their copyright files, and their dpkg statuses, along with some symlinks and a go binary for switching between iptables-legacy and iptables-nft. That image should be smaller and have less vulnerability scanner alerts. Step 1) is probably reasonable to consider regardless of step 2) anyhow, if someone wanted to take this up. You'll need to talk to SIG Network about these changes first though. This new image would also be a breaking change for any other projects that base on debian-iptables.
/milestone v1.25 kick this down the road, no activity.
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
the release 1.25 bug triage Lead. I'm reaching out to see the status of this Issue and if we are still targeting the current release.
there was progress (please see ), we will have to take an inventory when 1.26 opens up.

/milestone v1.26
Heya and - 1.26 bug triage team checking in to see how we're looking for this release? Thanks!
the last thing we wanted to do here was distroless kubeproxy which got done already. We should close this out! thanks everyone!
Do you mind doing a quick update to and to reflect the fact that we now have kubeproxy with distroless images? thanks!
We should probably still make distroless kubeproxy lighter and make sure our projects pick it up so we can phase out the old full image, do we have follow-up issues for getting the shell etc out of there and making sure we've rolled out the image to subprojects?


Question: } // DeletePodWithWait deletes the passed-in pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWait(c clientset.Interface, pod *v1.Pod) error { if pod == nil { return nil

Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not?

more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)


Question: // Delete ClusterIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) <ins> // Delete ExternalIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) // Delete LoadBalancerIP entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc) </ins> // Delete NodePort entries fcmd.CombinedOutputScript = append(fcmd.CombinedOutputScript, cmdFunc) fexec.CommandScript = append(fexec.CommandScript, execFunc)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --UDP traffic to a Pod behind a LoadBalancer Service exposing a UDP listener service is blackholed when the replacement Pod is scheduled on the same node as the "old" Pod. A new Pod that gets provisioned (replacing the previous one) will receive traffic once it's ready. a UDP Listener Pod (single pod) and using nodeSelector ensure it runs on a specific node the Pod behind a LoadBalancer Service continuous stream of UDP packets from outside the cluster to the External IP of the Service that traffic is correctly received the Pod and observe as a new one is created, traffic should now be blackholed and not reaching the "new" Pod After the new Pod is recreated, we noticed the following entries on the node: notice that the <SVC EXT IPappears here both in the original and the NATed , this is wrong and I suspect happens because the conntrack for the old Pod were flushed before the iptables rules for the new Pod were created (?) and in the interim new packets have reached the node and resulted in these incorrect conntrack entries. To "fix" this, we manually deleted the above conntrack entries, and this allowed new correct one to be established based on the now existing iptables nat rule: with this new conntrack entries, the traffic started flowing as expected to the new Pod and notice that the NATed now correctly shows the Pod IP (172.30.3.219) instead of the Service External IP. While spelunking through the GitHub issues and PRs, I found this issue which was addressed in , it does mention that: the service has endpoints available. Since the fix in this PR refers to NodePort, perhaps the same also applies to External IP (i.e. LoadBalancer Service?) or there's something else at play here? AFAICT, this fix is included in v1.20.7 that we are running. Kubernetes version (use ):Cloud provider or hardware configuration: GCP/GKE OS (e.g: ):Kernel (e.g.

): Install tools: Terraform (with google provider) Network plugin and version (if this is a network-related bug): Others:
/sig network
According to GCP support (since we hit this on a GKE cluster) there's already a fix for this bug and it in v1.21, however I'm unable to find to link it here (hope they can provide the link shortly).
GCP support referenced as the fix for this issue, however this was already merged and released in v1.20.4 but we are running v1.20.7, so should have the fix already, so the status is unclear at this point.
thanks for the detailed report, at first sight we do clean the external ips is it possible to set kube-proxy in a log level of 2 and capture the logs from the node to correlate with the time it happends, and understand why those entries are not being deleted?
Hey we can try that, however since this is a GKE environment I'd have to check how/if we can change kube-proxy log level. My understanding of the issue is that it's not that the conntrack entries are flushed, is that they are flushed before the nat iptables rules are created, causing faulty conntrack entries to be created in the interim (since the udp packets are incoming all the time).
I'm having some miscommunication about the availability of fix in with GCP/GKE support and was hoping you could confirm that is included in v1.20.5 (as per the release notes there? ) and per the cherry-pick PR GCP Support claim that this change is incorporated in v1.21 and we should wait for v1.21 to be available in GKE.
I see the commit in 1.20.5 , this is the cherry-pick The conntrack entries are always deleted after the conntrack entries are created, there is some code that detects that a service goes stale and based on that it deletes the entries.

I think that current code detects this case, at least I know it works when the service goes from 0 to 1 endpoint, that is why the logs will be useful, we can see exactly what happens ...
Thanks for confirming I just checked and seem to be already running in log level 2: I looked back at the logs when this issue was identified and I see the following entries: This two messages and only these two repeat every time we tried to rollout the deployment. is indeed the and is the ClusterIP of the service: but every time we tried, we ended up with these stale conntrack entries I mentioned in the OP and only after deleting them would the udp packet start flowing into the "new" pod.
oh wait, by external IP you mean ?
/triage accepted /assign
here you go, having good reports on bugs make them easy to spot ;)
indeed! I apologize if I wasn't clear indeed I was referring to the LoadBalancer Service External IP. Thanks for fix!!! But now I'm a bit confused, since I understood that LoadBalancer Service External IP == LoadBalancer Ingress IP, but I see these are two separate attributes? so for example: These are the same IP, but internally they are separate attributes?
hmm I don't know how kubectl prints the information, but you can check the fields with and you could see that ExternalIPs and LoadBalancer.IngressIP are different fields, actually one is in the Status and other in the Spec
interesting, so in the output, I can only see the , there's actually no mention of here: That's why I always assumed that this is referring to the same thing/attribute/field.


Question: copy_dev_tarballs() { echo "cluster/rackspace/util.sh: Uploading to Cloud Files" <del> ${SWIFTLY_CMD} put -i ${RELEASE_DIR}/kubernetes-server-linux-amd64.tar.gz  </del> <ins> ${SWIFTLY_CMD} put -i ${SERVER_BINARY_TAR}  </ins> ${CLOUDFILES_CONTAINER}/${CONTAINER_PREFIX}/kubernetes-server-linux-amd64.tar.gz > /dev/null 2>&1 echo "Release pushed."

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: }, metav1.UpdateOptions{}); err != nil { return errors.Wrap(err, "unable to update the CoreDNS ConfigMap") } <ins> // Point the CoreDNS deployment to the `Corefile-backup` data. if err := setCorefile(client, "Corefile-backup"); err != nil { return err } </ins> fmt.Println("[addons] Migrating CoreDNS Corefile") changes, err := migration.Deprecated(currentInstalledCoreDNSVersion, kubeadmconstants.CoreDNSVersion, corefile) if err != nil {

What happened: Coredns failed to start in 1.17 deployment when it tried to retain a backed up Corefile with error message: Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory What you expected to happen: Coredns starts normally with the backed-up Corefile How to reproduce it (as minimally and precisely as possible): Deploy a k8s 1.17 cluster and trigger coredns Corefile migration and make it fail. In our case, this was triggered in a three master -setup with 'old' method of running 'kubeadm init' for each master. Also the core-dns docker image used was tagged with '1.6.7-1', apparently causing the corefile migration version check to fail Anything else we need to know?: This seems to be caused by Both 'key' and 'path' are set to 'Corefile-backup' when this function is called. Should the 'path' always stay as 'Corefile' because that is where the container expects to find the config file? Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:51:57Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:47:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Custom deployment in Openstack environment OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" Kernel (e.g. ): 12.14-197.29-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/assign
/area kubeadm


Question: } } <ins> for _, ip := range service.Spec.PublicIPs { if ip == "0.0.0.0" { allErrs = append(allErrs, errs.NewFieldInvalid("spec.publicIPs", ip, "is not an IP address")) } else if util.IsValidIP(ip) && net.ParseIP(ip).IsLoopback() { allErrs = append(allErrs, errs.NewFieldInvalid("spec.publicIPs", ip, "publicIP cannot be a loopback")) } } </ins> return allErrs }

When spinning up VMs w/ kube, I attempted to use but this fails. ironically, 127.0.0.1 succeeds (internally you can then access a kube service via , for example, but you cannot do this if you try to bind to 0.0.0.0. binding works to 127.0.0.1, but is a horrible idea (since this IP wont be reachable by the outside world), so in that case kube should print a clear message on why this is the case and fail. binding to 0.0.0.0 fails right out of the bat, so in this case, also, kube should fail immediately. To be clear, im suggesting that these should fail at the service submission layer. Possibly a module for validating logic of submitted services/pods , which extends syntactical validation is in order if one doesn't already exist?
Please also see
Since this is very similar to I'm gonna take this.


Question: return &azureToken{ token:       spt.Token(), <ins> environment: token.environment, </ins> clientID:    token.clientID, tenantID:    token.tenantID, apiserverID: token.apiserverID,

What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var _ = Describe("Docker Containers", func() { var c *client.Client <ins> var ns string </ins> BeforeEach(func() { var err error c, err = loadClient() <ins> ns_, err := createTestingNS("containers", c) ns = ns_.Name </ins> Expect(err).NotTo(HaveOccurred()) }) <ins> AfterEach(func() { if err := c.Namespaces().Delete(ns); err != nil { Failf("Couldn't delete ns %s", err) } }) </ins> It("should use the image defaults if command and args are blank", func() { <del> testContainerOutput("use defaults", c, entrypointTestPod(), []string{ </del> <ins> testContainerOutputInNamespace(ns, "use defaults", c, entrypointTestPod(), []string{ </ins> "[/ep default arguments]", }) })

Testing how kubelet client rotation works, I discovered that on startup kubelet uses the cert in () and not the new cert in () This prevent kubelet to start kubelet uses the latest client certificate a 1 node cluster with kubespray (kubeadm should do) ntpd/chronyd the date 11 months in the future master certs kubelet that you have a new client cert the date 11 months more in the future and restart kubelet kubelet fails to start because it uses the cert embedded in instead of Both and are set, if I remove kubelet boots but fails to renew the cert Here an easy fix: Also I noticed that is RSA where is EC/P-256 Kubernetes version (use ): v1.18.15 (I can retest in more recent version but haven't seen anything in the release notes) Cloud provider or hardware configuration: VM OS (e.g: ): Kernel (e.g. ): Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle (not sure if this is more kubelet or kubeadm)
is the embedding certificates? see the second warning under here:
you can call on the same node to make the point to the rotatable file. newer versions of kubeadm do this change automatically.
i don't see a core bug here. i missed that you are using kubespray, which uses kubeadm. please report any issues with the kubespray issue tracker first: /kind support /close
Closing this issue. <detailsIn response to : i don't see a core bug here. please triage any issues with the kubespray issue tracker first: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks kubespray indeed doesn't call anywhere, I'll go look at how to fix that
like i said, newer versions of do this automatically.

this might be the result of an old cluster being upgrade using but the is remaining stale. the manual step in or calling the on a node is required.
It was a fresh deployment of k8s 1.18 / kubespray 2.15, I'm doing a fresh deployment of k8s 1.20 / kubespray master right now, but maybe kubespray is using kubeadm in weird ways / include old workarounds that leave us with having embedded certs instead of pointing to .
Found the culprit: Kubespray regenerate the conf files using kubeadm to make sure the server is up to date Thanks for the help and sorry for the spam
it would make sense to just include the kubelet-finalize phase in there too.
I prefer to just change the server using one of lineinfile/sed/kubectl so next time there is a new step like kubelet-finalize we don't break again.


Question: package v1beta1 import ( <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" )

no explanation for metadata filed of NodeMetrics/PodMetrics, code: should be like $ kubectl explain NodeMetrics KIND: NodeMetrics VERSION: DESCRIPTION: NodeMetrics sets resource usage metrics of a node. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. usage <map[string]string-required- The memory usage is the memory working set. window <string-required- and $ kubectl explain PodMetrics KIND: PodMetrics VERSION: DESCRIPTION: PodMetrics sets resource usage metrics of a pod. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: containers <[]Object-required- Metrics for all containers are collected within the same time window. kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. window <string-required- able to see the explanation for metadata filed of NodeMetrics/PodMetrics Kubernetes Version: v1.22.0-rc.0+
/sig instrumentation
/kind documentation /assign /triage accepted


Question: "//vendor/github.com/miekg/dns:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", <ins> "//vendor/github.com/pkg/errors:go_default_library", </ins> "//vendor/google.golang.org/api/compute/v1:go_default_library", "//vendor/k8s.io/utils/net:go_default_library", ],

What happened: The iperf tests were written a while ago but i think havent been updated. They measure bandwidth over multiple pods, but they probably should also have a test which reports individual pod's bandwidth between one pod over a SEP. They should pretty print a table so its easy to see exactly what is going on with the bandwidth They should use iperf3 (need to check that) Maybe we should have permutations for udp and tcp? The current output is something like: What you expected to happen: That running the tests would result in a table such as those that you can get from iperf3, i.e. How to reproduce it: run the e2es on any cluster, Environment: Kubernetes version (use ): master, any version
/sig network
/assign
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I've taken a first pass at this , working on resolving a couple issues then we can circle back to determine how else it could be improved!
Cool c/p the new test output so we can compare when u get a chance
ah yess. much better then what we have now.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <del> 1.29 </del> <ins> 1.29-1 </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted


Question: t.Fatalf("unexpected response: %v %t %#v", err, singular, test.Infos) } <del> info := test.Infos[1] if info.Name != "redis-master" || info.Namespace != "test" || info.Object == nil { </del> <ins> info := test.Infos[0] if _, ok := info.Object.(*api.ReplicationController); !ok || info.Name != "redis-master" || info.Namespace != "test" { t.Errorf("unexpected info: %#v", info) } info = test.Infos[1] if _, ok := info.Object.(*api.Pod); !ok || info.Name != "nginx" || info.Namespace != "test" { </ins> t.Errorf("unexpected info: %#v", info) } }

It works when I pipe but not with filename.
broke in this commit cc
Did you try to add an extension .json to the file? We filter for file extensions
ahhh ok. so working as intended. it used to autodetect :( ... thanks
That was unintended. The file extensions should only be used for deciding which files in a directory to process. For a single file, any extension should be allowed. We don't actually infer the content type from the extensions. For stdin and http sources, for instance, there are no extensions. cc
Fix is pretty straightforward (ignore file extensions when dealing with single file); I have it working with manual test, just adding unit test. Will send shortly.
Thanks, but this is v1.0-post, so no rush


Question: "203.0.113.1", "podFoo", "domainFoo", <ins> []v1.HostAlias{}, </ins> `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? Ingress, hosts and /etc/hosts Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request There have already been a few issues regarding modifying : , But both have a different use-case. First some information before I explain my use-case. I have a pod which run a web app (nginx container + php-fpm container) and it available over multiple sub domains. I want request to the exposed sub-/domains (ex. ) to go directly to the local webserver instant of going the long way (Cloudflare -Ingress Controller -Node -Container). There is 2 reason behind this: is faster/reduce overhead. of our app code call the exposed domains, and need to hit the same server, as the request oriented from (the code write to some tmp file, and other PHP code need to read that tmp file, when it get a request. Bad design: properly). cc Kubernetes version (use ): Environment: DigitalOcean, CoreOS.
I'm not sure I understand the use case, especially the 2nd. This is the case of a pod talking to itself via the public hostname? Why not just use localhost?
Public hostname == ingress . Yes, without going through the load balancer, some of the PHP also require "sticky" sessions and then it is just easier if the exposed domains just point to localhost. That require extra logic in the app code and when the app code expose multiple sub-/domains, then it is just easier adding the domain to /etc/hosts.
Do you know all the hostnames at pod startup time, or can they possibly be later? I guess I am not against this idea in principle, I just have a nagging suspicion it will blow in all the corner cases. I guess we could look at a community implementation, if someone did one. I don't think that we have time on our side right now to drive it.
I currently have a script like this: , which "extract" the hostnames from the nginx config at container startup (and "patch" the location of the file). It also configure to forward some request internally (we don't expose all domains externally).

-- I'm not sure about your particular use case as the host name extraction logic seems to be quite complex (parsing of the nginx config). The two things I do see that are useful: Ability for a ConfigMap to be written out as a single file into the container instead of injecting it as a directory of files. This would solve a lot of problems where a piece of the container configuration needs to be trivially changed. I'm fairly certain this is already filed, I can't seem to find the issue. Inheritance of host entries from the outer node.
I think the use-case beyond service is if the hosts need to be loopback or needs to be configured per pod. I tried to understand how is controlled by kubelet, and it seems like this feature would not be hard to implement. I'm willing to spend some time on this. in terms of implementation: from my reading, host mount is done by inside , and the content of the hosts file is controlled by the methods that call them all have access to the pod object. it seems like we can add a field to PodSpec (can be done as an Pod-level alpha annotation to start): or if we want something more explicit: that can be read into and , which then mounts as to the Pod.
Let me know if ^ makes sense. If so, I can make a formal proposal in or just a PR, whichever is appropriate.
I don't think this warrants a proposal or feature. I would argue for the more expliciti form of the API, but hostnames has to be plural, and IP has to be the mergeKey. or - any objections? Consider from a CRI point of view?
For me, this solves my problems.
so the struct would be something like where the keys are IP addresses, and values are hostnames. I presume this field would live under PodSpec. is the key we want to surface? ? the content would be: which after some merging would look like: (the go struct represented in yaml) and the output could either look like: or I think the latter is better since it's one line per entry, and seems to follow existing convention in .
Your proposal can definitely work for my use-case, although I would love if I could configure it from a init container, but this could work.

Keep up the good work :)
I wouldn't try to massage it into a map, just: , Kristian Klausen
So you'd be passing in into . That makes sense. This might be a stretch, but what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP.
This is currently handled completely by kubelet, so it wouldn't be an issue for CRI. kubelet doesn't mount if the pod is using the host network, so this would either need to be changed, or validation should reject the pod if this is not allowed. If you need to dynamically change the file after the container starts, I think suggestion of trying to make ConfigMap work would be better.
I think validation to reject the option if the pod is using the host network makes sense. is still a static file though. And if you inject a file that replaces , what are you going to do with existing dynamic hostnames that are by kubelet?
I don't have any concern from CRI perspective, and the proposal look fine to me from the high level. what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP. But can I request not stretching the feature to support dynamic changes? Pod can be deleted, and re-created, thus the pod's IP might be changed. Currently we only allow container image being updated in PodSpec. Unless we allow updating the new field, let's call it etcHost for now, in PodSpec, it won't work. Even we allow that field being updated, I think that is overkill.
With regards to the dynamicness, I'm just trying to design the API in such a way that's flexible. The field in PodSpec won't be dynamic. I imagine something like this: (The idea is borrowed from ) validation ensures that is only present if type is . then IP is gathered at Pod creation time. In kubelet's :
Why do you need pod IP? Why not use 127.0.0.1 ?

, Ricky Pai <:
The use-case that i'm more interested in is support for nodeIP, but I'm using podIP as an example because it's easier to illustrate with current code. But yes I agree with you that pod IP might not make sense since you have access to localhost already. I can imagine nodeIP being used to talk to daemons that are shared across pods. is a proper fix for the use-case, but this could be one way to address it.
sorry for the scope-creep. I'm just afraid using IP as a string field makes it hard to add support for other IP sources should that ever be an option. but it's probably premature right now. for now, I'll just proceed with:
Given that we have a way to get node IP through downward API, why do we need that here? On Apr 13, 2017 6:09 PM, "Ricky Pai" <:
Ideally, if the applications running inside the containers have flexible ways to configure dependent hostnames like through environment variables, then downward API would be sufficient. We can probably extend the argument that the whole support for is moot if applications supported flexible injection of dependent hostnames. The problem is many applications don't provide this level of flexibility. Hostnames are often hard-coded inside config files and the client libraries. Outside of Kubernetes, I might use a script to manage and expect nothing else to change the file; while it is possible that I use an init container to do the same modification to , since the mounting of happens whenever a container is restarted, I risk the file being changed and my application not being able to connect to its dependent services. This creates much headache migrating applications from a more lenient environment to Kubernetes, which is what we've been facing. I think support is the right direction and is why I'm willing to commit time to make it happen.
We have another issue open for "node-local Services" which sounds like what you want. I am not a big fan of the expansion of host mappings like this. I'd rather go all the way and give nodes a static link-local address or else solve node-local services. , Ricky Pai <:
Thanks for all the feedback I'm well aware of node-local services, but it seems like progress on the proposal is stalled.

I believe node-local services would be the proper way to address the use case I described, especially with the "locality" concept you described in that can be expanded beyond the node. However, in lieu of node-local services, with references to the node IP might be a quick way to address the use case. Other tickets describing the need for node-level sidecars also proposed either using node-local services or : I also understand this complicates the interface a little bit more, but it would really benefit cluster operators. If you still think this is inappropriate, then I can proceed with this interface next week:
I don't have a strong opinion on this topic but it seems that it should be addressed as part of what is the status of the proposal?
I am not familiar with the node-local services and its progress, but if this can satisfy the use cases, I'd suggest that not adding new fields to the pod API for a quick workaround.
I feel the original proposal got digressed by my zealous talk of node IP support. Allow me to clarify: serves several use cases: external hostname with a specific IP. i.e. if I want to say mock out an external hostname like a hostname with localhost. i.e. if I want to override to point to we allow dynamic injection of nodeIPs, it can be an easy way for pods to talk to nodes in absence of node-local services. only the third use-case is relevant to node-local services. I think it's still useful to support adding entries to even with static IPs
I understand the third case, but I'd like to consider other options to solve that, first. I think the rest is uncontentious. , Ricky Pai <:
Thanks. I'll get the changes up this week. There'll likely be three changes: tests for existing hosts file content (there's no tests currently) change for on to use in Is there any possibility to cherry-pick those back in 1.6? Or would this have to wait for 1.7?
AFAIK, we don't cherrypick features.
has been created for the API change
just to clarify, are you in favor of adding HostMappings to the API?
I am neutral on it. It's come up often enough that i believe the problem is real. The API doesn't seem harmful to me, what do you think?

You're the guardian of CRI... , Yu-Ju Hong <:
Also the name is terrible - help? HostAliases? DNSAliases? , Tim Hockin <: solve that, first. I think the rest is uncontentious. , just to clarify, are you in favor of adding HostMappings to the API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I'm up for . might accidentally imply that it's actually interfacing with kube-dns.
I am neutral but just wanted to make sure there are no obvious, alternative ways to achieve this. BTW, the proposed solution changes the k8s API, but not CRI :) On the other hand, if we support this on the k8s API level, we should discourage users from modifying this file directly. For example, reported a bug where users modify in the init containers, and later found that kubelet overwrites it.
We should definitely discourage modifying that, but we should perhaps respect it if a user provided a volume mount. Thoughts on name? , Yu-Ju Hong <:
I am terrible at naming... I agree that having DNS in the name can be misleading. sounds ok. might be more direct?
I'd go for brevity when possible. HostAliases ? On Apr 27, 2017 5:27 PM, "Yu-Ju Hong" <:
I'm updating the PR to use . thanks for the suggestion
Just posted the last PR to enable this feature:
could you add a node e2e test for this feature?
Do we need e2e? Every e2e is a liability, and slows down test runs - e2e is needed if we can't cover in unit or integration. , Yu-Ju Hong <:
Unit tests are great to ensure the individual functions work correctly, but so far it's not sufficient to prevent regression in our codebase from my experiences. Since this is a new addition/feature to the API, I'd prefer having a corresponding node e2e test. Integration test can be sufficient, but I've not had any pleasant experience debugging integration tests. Maybe that has changed already?
no, integrtaion tests are under-serviced tech-wise.

, Yu-Ju Hong <:
does a simple node e2e test sound good to you?
SGTM , Yu-Ju Hong <:
I saw that the HostAlias's comment: I'm wondering why it couldn't work to hostNetwork pods? And how does a hostNework pod initialize its /etc/hosts file?
Are we still missing a test for this? I'd hate to have to kick it out of the 1.7 release for lack of tests..
yes we are still missing a test. what's the deadline for 1.7 release? sorry i was on PTO and dropped the ball
hostNetwork pods uses the host's file, while only non-hostNetwork pods have managed by kubelet. because kubelet does not manage the file when using hostNetwork, kubelet can't write additional entries from hostAliases.
Just chatted on slack, for containers using the host network, docker actually makes a copy of "/etc/hosts" on the host for each of them. I think kubelet can potentially support HostAliases for hostnetwork pods by copying and mounting the file. Not sure if there are be other concerns though.
if you make the change to make kublet copy and manage for hostNetwork pods, I can make the change for HostAliases. one concern: currently, the logic includes , including from and . Should this be allowed? if not, there should probably be validation disallowing those field when hostNetwork is used. there currently isn't any unless I read the code wrong.
node e2e test:
THANKS! , Ricky Pai <:
That would be a great feature! Currently we have pods using hostNetwork and using init containers to restore some host mappings -- basically . With HostAliases feature it would be fantastic fit!
We are using and for hostNetwork pods.
This is something I don't understand in the code: It seems that it is still mounting "/etc/hosts" for hostnetwork pods if . Does that mean kubelet is still managing it?
One of our current assumptions is that kubelet will mount a shared /etc/hosts (which is not node's) for all containers within a pod in hostNetwork mode. Which seems to be the codepath in the code linked above. Is this holding true?
I checked the git history and that was just an artifact caused by switching from internal to v1 API objects.

The original code is: In the internal API, is a field in , so you can see why had to be non-nil. In the v1 API, is a field in the pod spec: . When switching all code to use the versioned (v1) objects, this got lost in translation. I believe the apiserver sets the default pod SecurityContext, so it should never be nil. I'll send a PR to remove the redundant check.
Thanks for looking into that. Currently even with hostNetwork pods, it will have mounts for . Checking within the pod: If the check is removed, what would happen on the shared mount of /etc/hosts ? My concern and question is about the guarantee here. Is there still a copy of which is shared and mounted for all containers within a pod?
Removing the check itself won't affect anything since securitycontext should always be non-nil. That mount for hostnetowrk pods was by docker (not by kubelet) so it should be one per container, and not shared by containers within a pod. I didn't try it myself, but it should be easy to verify.
Tried and verified that was shared between containers within a pod for k8s 1.6.4 .
Also verified that modifying for a hostNetwork pod did not change the file on the node.
Then docker did the right thing and use the same file for containers sharing the same network namespace :-)
That makes sense.
Will HostAlias support for HostNetwork pod happen in 1.7 ?
I'm trying to get in so it can make the 1.7 release deadline. help reviewing the PR would be great.
in a multi-container hostNetwork pod, what happens when you change the file, then restart one of the containers? in non-hostNetwork pod, it would revert the changes to
anyways, i'm also in favor of kubelet managing the file even for hostNetwork pods. that way, we can use features like hostAliases, and also then we don't depend on the container runtime to dictate the behavior of the hosts file. it's not clear whether running rktnetes with hostNetwork pod would yield the same result
Is this a dup of ? How are they different?
HostAlias is useful to provide local overrides for a particular Pod, but it seems like would want the alternative DNS name of a service accessible for all Pods.

re: documentation, i'm thinking either adding notes about HostAliases in or as a separate page under "Services, Load Balancing, and Networking"
Is there any issue created for supporting HostAlias in hostnetwork mode?
I can write up what I think will be needed to support HostAlias in hostNetwork pods
This is now in 1.7:
filed issues to follow up with HostNetwork support:
Thank you for this feature. Our use case is different. On an nginx/php-fpm set up where our local dev is via docker-compose yet we're deploying into kubernetes, we can use the exact same nginx config on both environments. We're currently working around this via HOSTALIASES (alias or or whatever the php container is into , which satisfies both deployment types), but this is hacky and doesn't work in Alpine.
the discrepancy between dev/prod would probably cause problems beyond just difference in networking setup. have you considered using to have parity between the two environments?
We have, and found it way too cumbersome for local development. It's way heavier in resources as well - obviously there's a lot more than just docker needed for minikube to run, and most of our developers use Mac, where all this is virtualised. In contrast, docker-compose is a lot easier, more concise, and we have yet to find any environment-y issues caused by difference between these two, except for this one issue we fortunately have a workaround for (HOSTALIASES). We use the exact same base container images for development as well as builds. From the point of view of the services themselves, there's no difference at all being run on one environment or the other - they talk to each other via HTTP or queues.
that's unfortunate. I guess minikube does have overhead since you have to run it inside a VM on your mac machines. But since all components are written in go and cross-compiled in darwin, I wonder if you could setup your own etcd (homebrew has this), then your own kubelet. Docker you can run Docker on Mac, then just have kubelet talk to the local docker socket. This sounds like it would work with lower resource overhead than minikube since you don't to run in a VM. Either way, I'm glad HostAlias is able to address your problem.

I just wish you didn't have to deal with this discrepancy in the first place.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: // The failure information is given by the error. type FitPredicate func(pod *v1.Pod, meta Metadata, nodeInfo *schedulernodeinfo.NodeInfo) (bool, []PredicateFailureReason, error) <del> func isVolumeConflict(volume v1.Volume, pod *v1.Pod) bool { // fast path if there is no conflict checking targets. if volume.GCEPersistentDisk == nil && volume.AWSElasticBlockStore == nil && volume.RBD == nil && volume.ISCSI == nil { return false } for _, existingVolume := range pod.Spec.Volumes { // Same GCE disk mounted by multiple pods conflicts unless all pods mount it read-only. if volume.GCEPersistentDisk != nil && existingVolume.GCEPersistentDisk != nil { disk, existingDisk := volume.GCEPersistentDisk, existingVolume.GCEPersistentDisk if disk.PDName == existingDisk.PDName && !(disk.ReadOnly && existingDisk.ReadOnly) { return true } } if volume.AWSElasticBlockStore != nil && existingVolume.AWSElasticBlockStore != nil { if volume.AWSElasticBlockStore.VolumeID == existingVolume.AWSElasticBlockStore.VolumeID { return true } } if volume.ISCSI != nil && existingVolume.ISCSI != nil { iqn := volume.ISCSI.IQN eiqn := existingVolume.ISCSI.IQN // two ISCSI volumes are same, if they share the same iqn. As iscsi volumes are of type // RWO or ROX, we could permit only one RW mount. Same iscsi volume mounted by multiple Pods // conflict unless all other pods mount as read only. if iqn == eiqn && !(volume.ISCSI.ReadOnly && existingVolume.ISCSI.ReadOnly) { return true } } if volume.RBD != nil && existingVolume.RBD != nil { mon, pool, image := volume.RBD.CephMonitors, volume.RBD.RBDPool, volume.RBD.RBDImage emon, epool, eimage := existingVolume.RBD.CephMonitors, existingVolume.RBD.RBDPool, existingVolume.RBD.RBDImage // two RBDs images are the same if they share the same Ceph monitor, are in the same RADOS Pool, and have the same image name // only one read-write mount is permitted for the same RBD image. // same RBD image mounted by multiple Pods conflicts unless all Pods mount the image read-only if haveOverlap(mon, emon) && pool == epool && image == eimage && !(volume.RBD.ReadOnly && existingVolume.RBD.ReadOnly) { return true } } } return false } // NoDiskConflict evaluates if a pod can fit due to the volumes it requests, and those that // are already mounted. If there is already a volume mounted on that node, another pod that uses the same volume // can't be scheduled there. // This is GCE, Amazon EBS, ISCSI and Ceph RBD specific for now: // - GCE PD allows multiple mounts as long as they're all read-only // - AWS EBS forbids any two pods mounting the same volume ID // - Ceph RBD forbids if any two pods share at least same monitor, and match pool and image, and the image is read-only // - ISCSI forbids if any two pods share at least same IQN and ISCSI volume is read-only // TODO: migrate this into some per-volume specific code? func NoDiskConflict(pod *v1.Pod, meta Metadata, nodeInfo *schedulernodeinfo.NodeInfo) (bool, []PredicateFailureReason, error) { for _, v := range pod.Spec.Volumes { for _, ev := range nodeInfo.Pods() { if isVolumeConflict(v, ev) { return false, []PredicateFailureReason{ErrDiskConflict}, nil } } } return true, nil, nil } </del> // MaxPDVolumeCountChecker contains information to check the max number of volumes for a predicate. type MaxPDVolumeCountChecker struct { filter         VolumeFilter

/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


Question: path string } <ins> func NewFileStore(path string) (CheckpointStore, error) { if err := ensurePath(path); err != nil { return nil, err } return &FileStore{path: path}, nil } </ins> func (fstore *FileStore) Write(key string, data []byte) error { if err := validateKey(key); err != nil { return err } <del> if _, err := os.Stat(fstore.path); err != nil { // if directory already exists, proceed if err = os.MkdirAll(fstore.path, 0755); err != nil && !os.IsExist(err) { return err } </del> <ins> if err := ensurePath(fstore.path); err != nil { return err </ins> } tmpfile := filepath.Join(fstore.path, fmt.Sprintf("%s%s%s", tmpPrefix, key, tmpSuffix)) if err := ioutil.WriteFile(tmpfile, data, 0644); err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: "testing" "time" <del> "k8s.io/klog/v2" </del> "github.com/stretchr/testify/assert" v1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/klog/v2" </ins> "k8s.io/kubernetes/pkg/proxy" "k8s.io/kubernetes/pkg/proxy/healthcheck" utilproxy "k8s.io/kubernetes/pkg/proxy/util"

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: } func PodToleratesNodeTaints(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") } taints, err := v1.GetTaintsFromNodeAnnotations(node.Annotations) </del> <ins> taints, err := nodeInfo.Taints() </ins> if err != nil { return false, nil, err }

Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though.

This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.


Question: cd easy-rsa-master/easyrsa3 ./easyrsa init-pki > /dev/null 2>&1 <del> ./easyrsa --batch build-ca nopass > /dev/null 2>&1 </del> <ins> ./easyrsa --batch "--req-cn=$cert_ip@`date +%s`" build-ca nopass > /dev/null 2>&1 </ins> if [ $use_cn = "true" ]; then ./easyrsa build-server-full $cert_ip nopass > /dev/null 2>&1 cp -p pki/issued/$cert_ip.crt "${cert_dir}/server.cert" > /dev/null 2>&1

When you generate your second cluster, suddenly you'll find that nothing works from browsers. The ca/cert pairs we make reuse serial numbers. Firefox says: Chrome just refuses to show anything, which made this hard to debug. We generate the ca & cert here: Action items: Every CA we make should be unique, even within the same (e.g.) GCE project/ip address. Every cert we make should have a unique serial number. easyrsa docs don't make it immediately obvious how to accomplish this.
running build-ca in --batch mode wipes out the existing pki, which restarts the indexing at 0 if the ca already exists, shouldn't we just skip build-ca?
I will happily merge a PR with that change! I only understand this stuff well enough to figure out what's wrong... ;)
Actually--if that is a/the problem, then running kube- should bork my cluster, no? But actually it does seem to work. So I suspect that the problem is deeper than that, like we generate the same ca deterministically (or some important component thereof) when kube- runs? Like all of the browsers on my desktop now completely refuse to talk to any kubernetes master because of this.
Ah, it looks like we end up with this in all certs: ... I think we want to add a arg to the call to uniquify the certificate authority each time it's generated
That sounds like it's probably the problem. Thanks!
fixes this.


Question: CURRENT_VERSION=${step} echo "${CURRENT_VERSION}/${CURRENT_STORAGE}" > "${DATA_DIRECTORY}/${VERSION_FILE}" fi <del> if [ "$(echo ${CURRENT_VERSION} | cut -c1-2)" = "3." -a "${CURRENT_STORAGE}" = "etcd2" -a "${TARGET_STORAGE}" = "etcd3" ]; then </del> <ins> if [ "$(echo ${CURRENT_VERSION} | cut -c1-2)" = "3." -a "${CURRENT_VERSION}" = "${step}" -a "${CURRENT_STORAGE}" = "etcd2" -a "${TARGET_STORAGE}" = "etcd3" ]; then </ins> # If it is the first 3.x release in the list and we are migrating # also from 'etcd2' to 'etcd3', do the migration now. echo "Performing etcd2 -> etcd3 migration"

At the very least it should be clear when (timestamp) the migration started and finished.

If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.


Question: local portable_tarball_name="${RELEASE_TARS}/kubernetes-test-portable.tar.gz" kube::release::create_tarball "${portable_tarball_name}" "${release_stage}/.." <del> if [[ "${KUBE_BUILD_MONDO_TEST_TARBALL}" =~ [yY] ]]; then kube::log::status "Building tarball: test mondo (deprecated by KEP sig-testing/20190118-breaking-apart-the-kubernetes-test-tarball)" local platform for platform in "${KUBE_TEST_PLATFORMS[@]}"; do local test_bins=("${KUBE_TEST_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then test_bins=("${KUBE_TEST_BINARIES_WIN[@]}") fi mkdir -p "${release_stage}/platforms/${platform}" # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # test_bins array. cp "${test_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/platforms/${platform}" done for platform in "${KUBE_TEST_SERVER_PLATFORMS[@]}"; do mkdir -p "${release_stage}/platforms/${platform}" # This fancy expression will expand to prepend a path # (${LOCAL_OUTPUT_BINPATH}/${platform}/) to every item in the # KUBE_TEST_SERVER_BINARIES array. cp "${KUBE_TEST_SERVER_BINARIES[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/platforms/${platform}" done cat <<EOF > "${release_stage}/DEPRECATION_NOTICE" The mondo test tarball containing binaries for all platforms is DEPRECATED as of Kubernetes 1.14. Users of this tarball should migrate to using the platform-specific tarballs in combination with the "portable" tarball which contains scripts, test images, and other manifests. For more details, please see KEP sig-testing/20190118-breaking-apart-the-kubernetes-test-tarball. EOF kube::release::clean_cruft local package_name="${RELEASE_TARS}/kubernetes-test.tar.gz" kube::release::create_tarball "${package_name}" "${release_stage}/.." fi </del> } # This is all the platform-independent stuff you need to run/install kubernetes.

What happened: On 2019-02-22 was merged with an indication that the mondo test tarball noted as deprecated as per But it did not include a timeline for eventual removal. What you expected to happen: It's been 7 months now. We should formalize a plan for removal. /sig testing /sig release /kind cleanup /priority important-soon
Enhancement tracking issue ref:
removes the deprecated mondo test tarball.


Question: API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,MicroTime,Time API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,StatusCause,Type API rule violation: names_match,k8s.io/apimachinery/pkg/apis/meta/v1,Time,Time <del> API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,RawExtension,Raw </del> API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentEncoding API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,ContentType API rule violation: names_match,k8s.io/apimachinery/pkg/runtime,Unknown,Raw

The Service Catalog includes the type in . The openapi schema generated for it is wrong. It has two problems: it includes the field, which is marked as required it doesn't allow additional properties to be included under it Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If the Service Catalog openapi schema is exposed to kubectl (this currently requires a few tweaks and a patch), it reports validation errors when creating a ServiceInstance resource. What you expected to happen: The schema should allow to include arbitrary properties and not require the property to be included. How to reproduce it (as minimally and precisely as possible): not easy at the moment (I'll add instructions later if I'm unable to fix this myself) Anything else we need to know?: I'd like to tackle this problem myself, but I need guidance. Should openapi-gen handle this automatically for all occurences of RawExtension, or should each field of type be json-tagged individually? Environment: Kubernetes version (use ):
I've heard you're the goto guy for the openapi-gen. Please help? :)
/sig api-machinery
OK, I've found I can specify a custom schema for RawExtension the way it's done in Time and MicroTime. The remaining problem is how to specify that the RawExtension object can be of any type. Using is not supported currently - if I use it in the schema, the API server dies with:
/assign /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
We've just hit this. /remove-lifecycle rotten /lifecycle frozen
hitting this
According to this: the raw extension can define its openapi type as: or alternatively what do you think?

do you still work on this or you want me to send a PR?
Would changing the definition of RawExtension to this help the situation? (adding ) I noticed that this removes Raw from the openapi defintion of it, without changing how it is marshalled or unmarshalled (since we have custom functions for that) I'll try making a PR for this


Question: select { case <-c: return <del> case <-time.After(2 * time.Second): </del> <ins> case <-time.After(wait.ForeverTestTimeout): </ins> t.Fatalf("Timeout while waiting for the plugin registration status") } }

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign