Question: exit 1 fi <ins> # Travis continuous build uses a head go release that doesn't report # a version number, so we skip this check on Travis.  Its unnecessary # there anyway. if [ ${TRAVIS} != "true" ]; then GO_VERSION=($(go version)) if [ ${GO_VERSION[2]} < "go1.2" ]; then echo "Detected go version: ${GO_VERSION}." echo "Kubernetes requires go version 1.2 or greater." echo "Please install Go version 1.2 or later" exit 1 fi fi </ins> pushd $(dirname "${BASH_SOURCE}")/.. >/dev/null KUBE_REPO_ROOT="${PWD}" KUBE_TARGET="${KUBE_REPO_ROOT}/output/go"

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /milestone v1.18 /priority critical-urgent /cc /sig cluster-lifecycle /area provider/gcp
/reopen Reopening, to observe the job.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Several of the recent failures appear to be this: cc
next run should pick up the change in we have had boskos outage clearing up now.
This is still not going to work because k/k has removed the .deb files we've been downloading from the CI artifacts:
FYI we are discussing in Slack how to proceed to resolve this. I doubt it will be a simple, quick fix, though.
me and did mention this on the k-dev: !msg/kubernetes-sig-cluster-lifecycle/q934Ah3SD64/JukDA9wsAgAJ given there were no replies we assumed there are no users. but this is definitely for the better as we removed the Bazel build of packages from k/k. 's PR seems to use binaries directly: i will have a look.
Closed via There is a possibility we will publish debs again in the future (via ), but while that tool is in flux, I don't want anyone depending on it. Thanks for the fix, /close
Closing this issue. <detailsIn response to : Closed via /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

fails with this trace: Please help.
Thanks for trying Kubernetes, sorry it didn't work for you. Do you have an old version of go installed? We currently are building with go 1.2, you can run to find the version. I tried to repro. this on a clean client, and I can't seem to reproduce the problem, also our Travis build is green: https://travis-, so I think that the problem must be somewhere in your set up. Please let me know some more details and we can help you debug. Best! --brendan
I validated that this is due to trying to compile with Go version 1.0 rather than 1.2. I'll update the scripts to validate that the appropriate version of Go is installed. So please upgrade your installed version of golang, and try again. Thanks! --brendan


Question: // TODO: Use the addToNetwork function to obtain the IP of the Pod. That will assume idempotent ADD call to the plugin. // Also fix the runtime's call to Status function to be done only in the case that the IP is lost, no need to do periodic calls func (plugin *kubenetNetworkPlugin) Status(namespace string, name string, id kubecontainer.ContainerID) (*network.PodNetworkStatus, error) { <ins> plugin.mu.Lock() defer plugin.mu.Unlock() </ins> cidr, ok := plugin.podCIDRs[id] if !ok { return nil, fmt.Errorf("No IP address found for pod %v", id)

I gathered e2e test results of suite for last 100 runs. The following is the list of flaky tests. Some might not be related to . I will start with the service tests. Any help would be appreciated. Detail list: Jenkins project:
cc/ Cluster team is working on to make kubenet default. Once we switch this to production, we don't need to restart docker to configure cbr0.
After a bunch of PRs got merged, the kubenet suite is much more stable. One more known flake can be fixed by I will post new flake tests here.
Just bump into a new failure: One of the nodes () got panic: Looks like it crashed at shaper reset.: Can you take a look?


Question: var ( gpuResourceName v1.ResourceName <del> dsYamlURL       string </del> ) func makeCudaAdditionDevicePluginTestPod() *v1.Pod {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .


Question: RollbackTemplateUnchanged = "DeploymentRollbackTemplateUnchanged" // RollbackDone is the done rollback event reason RollbackDone = "DeploymentRollback" <del> // OverlapAnnotation marks deployments with overlapping selector with other deployments // TODO: Delete this annotation when we no longer need to support a client //       talking to a server older than v1.6. OverlapAnnotation = "deployment.kubernetes.io/error-selector-overlapping-with" </del> // Reasons for deployment conditions // // Progressing:

The Deployment overlap annotation () is no longer populated nor used by 1.6+ on the server side. However, in we maintained knowledge of the annotation for 1.6 clients in case they talk to pre-1.6 servers. That PR also adds code to clean up the annotation whenever it's found in 1.6+ clusters, so that pre-1.6 clients don't see it and make the wrong assumptions. The code handling this annotation should be removed when all the following conditions are met: We no longer support a 1.5.x client talking to the latest server version. We no longer support upgrading the server directly from 1.5.x to the latest version. The soonest milestone that meets both conditions according to the is v1.8. This issue should be moved to that milestone once it exists.
TODOs: [x] deployment controller: [x] deployment util: [x] deployment reaper: [x] deployment describer:
I can help on this
/assign
GitHub didn't allow me to assign the following users: wanghaoran1988. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks, feel free to send patches


Question: // IsNotFound returns true if the specified error was created by NewNotFound. // It supports wrapped errors. func IsNotFound(err error) bool { <del> return ReasonForError(err) == metav1.StatusReasonNotFound </del> <ins> reason, code := reasonAndCodeForError(err) if reason == metav1.StatusReasonNotFound { return true } if _, ok := knownReasons[reason]; !ok && code == http.StatusNotFound { return true } return false </ins> } // IsAlreadyExists determines if the err is an error which indicates that a specified resource already exists.

Checked a method error using when the error was caused by a validating webhook that returns a 403 response. In this particular case, the error was generated from a validating webhook implemented with controller-runtime that returns controller-runtime's response. to return true. returned false Many (but not all) of the other family of functions in suffer the same kind of problem. I have changes ready to submit in a PR if we can get agreement that this is a problem and that it should be fixed in the functions that ought to check the of an erorr value that is or wraps an Kubernetes version (use ): 19.3 for the server
/sig api-machinery
are you going to send a PR for this? The issue description looks reasonable. /triage accepted
Thanks, PR is submitted.
/assign


Question: // (e.g. the addition of a new field will cause the hash code to change) // Note that we assume input podTemplateSpecs contain non-empty labels func equalIgnoreHash(template1, template2 api.PodTemplateSpec) (bool, error) { <ins> // First, compare template.Labels (ignoring hash) labels1, labels2 := template1.Labels, template2.Labels </ins> // The podTemplateSpec must have a non-empty label so that label selectors can find them. // This is checked by validation (of resources contain a podTemplateSpec). <del> if len(template1.Labels) == 0 || len(template2.Labels) == 0 { </del> <ins> if len(labels1) == 0 || len(labels2) == 0 { </ins> return false, fmt.Errorf("Unexpected empty labels found in given template") } <del> hash1 := template1.Labels[extensions.DefaultDeploymentUniqueLabelKey] hash2 := template2.Labels[extensions.DefaultDeploymentUniqueLabelKey] // compare equality ignoring pod-template-hash template1.Labels[extensions.DefaultDeploymentUniqueLabelKey] = hash2 </del> <ins> if len(labels1) > len(labels2) { labels1, labels2 = labels2, labels1 } // We make sure len(labels2) >= len(labels1) for k, v := range labels2 { if labels1[k] != v && k != extensions.DefaultDeploymentUniqueLabelKey { return false, nil } } // Then, compare the templates without comparing their labels template1.Labels, template2.Labels = nil, nil </ins> result := api.Semantic.DeepEqual(template1, template2) <del> template1.Labels[extensions.DefaultDeploymentUniqueLabelKey] = hash1 </del> return result, nil }

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fi params+=" --tls-cert-file=/etc/srv/kubernetes/server.cert" params+=" --tls-private-key-file=/etc/srv/kubernetes/server.key" <ins> params+=" --requestheader-client-ca-file=/etc/srv/kubernetes/aggr_ca.crt" params+=" --requestheader-allowed-names=aggregator" params+=" --requestheader-extra-headers-prefix=X-Remote-Extra-" params+=" --requestheader-group-headers=X-Remote-Group" params+=" --requestheader-username-headers=X-Remote-User" params+=" --proxy-client-cert-file=/etc/srv/kubernetes/proxy_client.crt" params+=" --proxy-client-key-file=/etc/srv/kubernetes/proxy_client.key" params+=" --enable-aggregator-routing=true" </ins> params+=" --client-ca-file=/etc/srv/kubernetes/ca.crt" params+=" --token-auth-file=/etc/srv/kubernetes/known_tokens.csv" params+=" --secure-port=443"

We want kubemark setups to be as similar to real clusters as possible. Since we are launching kube-aggregator in real clusters, we should add a possibility (and probably also enable by default) to have kube-aggregator in front of apiserver(s).
was working on being able to use it in e2es.
With kube-aggregator integrated into kube-apiserver, is this still relevant?
There are a few flags which need to be set up for kubemark to actually be properly testing with the aggregator. I am digging around in that code this week anyway. I will try to get to it and make sure its set up properly.
Thanks for looking. Let me know if I you need something.
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details


Question: apiserver_dropped_requests_total{request_kind="mutating"} 1 # HELP apiserver_request_total [STABLE] Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code. # TYPE apiserver_request_total counter <del> apiserver_request_total{code="429",component="apiserver",dry_run="",group="",resource="pods",scope="cluster",subresource="",verb="POST",version="v1"} 1 </del> <ins> apiserver_request_total{code="429",component="apiserver",dry_run="",group="",resource="pods",scope="resource",subresource="",verb="POST",version="v1"} 1 </ins> `, }, {

I've observed, that some metrics has and labels. (create) requests are "single resource", therefor they should be classified as . Def: customresourcehandler uses , which is based on the RequstInfo only: the RequestInfo is based only on the requests: method and url example: e.g: Since this RequestInfo has Name and Namespace empty, it is classified as . In a similar way it could be explained why: There's a mismatch in scope classification between built-in and custom resources: The key difference - the check if either "name" was provided OR it is a : Where the metrics.CleanScope ONLY checks, if name is not-empty. Tested: = v1.20 There were other scope related issues <1.20 and the distinction is not obvious. <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: address            = flag.String("address", "127.0.0.1", "The address for the info server to serve on") port               = flag.Uint("port", 10250, "The port for the info server to serve on") hostnameOverride   = flag.String("hostname_override", "", "If non-empty, will use this string as identification instead of the actual hostname.") <ins> dockerEndpoint     = flag.String("docker_endpoint", "", "If non-empty, use this for the docker endpoint to communicate with") </ins> ) const dockerBinary = "/usr/bin/docker"

When using or with a key/value that has a multi-line string as value, that value will appear as quoted string in the log message, i.e. the log message will still be a single line. It would be nicer to print the value in text mode so that it goes across multiple lines, with clear indicators for start and end of the value. Indention could be used to avoid mistaking some embedded line as the start of a new log message. In JSON mode, the value should be formatted normally. /sig instrumentation /wg structured-logging Always printing as a single line can can make the log output unreadable ().
When adding something like this, can be reverted and implemented differently.
My original thinking was that only objects that explicitly ask for multi-line output get formatted that way. But that means that we need to review log calls very carefully. An alternative is to automatically switch from quoting strings to multi-line output based on some heuristic, for example when there's more than one line break.
Here's a potential implementation:
If structured logging is needed in any other case I can contribute, please inform if needed.
/triage accepted /assign /priority important-soon /area logging /cc /cc
We have the same problem as in kube-scheduler also in kubelet, it's just that no-one has seen and/or complained about the multi-line strings. With the benchmark + log analysis tool from I found the following multi-line string parameters in a kubelet log: <details</details


Question: } func (om *fakeObjectManager) CreatePod(ctx context.Context, pod *v1.Pod) error { <del> defer om.createPodTracker.inc() if om.createPodTracker.errorReady() { defer om.createPodTracker.reset() return om.createPodTracker.getErr() </del> <ins> defer om.createPodTracker.trackParallelRequests() if err := om.createPodTracker.incWithOptionalError(); err != nil { return err </ins> } pod.SetUID(types.UID(pod.Name + "-uid")) return om.podsIndexer.Update(pod)

https://prow.ppc64le- TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled of Have started observing more flakes recently since last week. No response_ Able to reproduce using stress tool. /sig testing /sig apps
/triage accepted /assign
opened a PR with a fix:


Question: "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", "--network-plugin=kubenet", "--pod-cidr=10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller. <del> "--hairpin-mode=hairpin-veth", </del> ) cmd := exec.Command("sudo", cmdArgs...) hcc := newHealthCheckCommand(

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: // the job stabilized and won't be synced until modification or full // resync happens, we don't want to wait for the latter so we force // sync modifying it <del> job.Spec.Parallelism = &completions job, err = updateV1Job(f.ClientSet, f.Namespace.Name, job) </del> <ins> _, err = framework.UpdateJobWithRetries(f.ClientSet, f.Namespace.Name, job.Name, func(update *batch.Job) { update.Spec.Parallelism = &completions }) </ins> Expect(err).NotTo(HaveOccurred()) err = waitForV1JobFail(f.ClientSet, f.Namespace.Name, job.Name, v1JobTimeout) }

https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}


Question: CLEAN_PATTERNS=( "_tmp" "doc_tmp" <del> ".*/zz_generated.openapi.go" </del> <ins> "((?!staging/src/k8s.io/apiextensions-apiserver/pkg/generated/openapi).)*/zz_generated.openapi.go" </ins> "test/e2e/generated/bindata.go" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to vendor , which due to (recently ?) code in depends on The problem now is that the content of is incomplete: apiextensions- calls generatedopenapi.GetOpenAPIDefinitions, but apiextensions-apiserver/pkg/generated/openapi/ doesn't contain the generated code. What you expected to happen: I expect to compile when fetching it with "go get" or vendoring it with tools like "dep". How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/assign
/sig release
/assign
sent a fix in


Question: resourceVersion string labelSelector   string fieldSelector   string <ins> namespace       string </ins> }{ { rawQuery:        "resourceVersion=1234", resourceVersion: "1234", labelSelector:   "", fieldSelector:   "", <ins> namespace:       api.NamespaceAll, </ins> }, { <del> rawQuery:        "resourceVersion=314159&fields=Host%3D&labels=name%3Dfoo", </del> <ins> rawQuery:        "namespace=default&resourceVersion=314159&fields=Host%3D&labels=name%3Dfoo", </ins> resourceVersion: "314159", labelSelector:   "name=foo", fieldSelector:   "Host=", <ins> namespace:       api.NamespaceDefault, </ins> }, { <del> rawQuery:        "fields=ID%3dfoo&resourceVersion=1492", </del> <ins> rawQuery:        "namespace=watchother&fields=ID%3dfoo&resourceVersion=1492", </ins> resourceVersion: "1492", labelSelector:   "", fieldSelector:   "ID=foo", <ins> namespace:       "watchother", </ins> }, { rawQuery:        "", resourceVersion: "", labelSelector:   "", fieldSelector:   "", <ins> namespace:       api.NamespaceAll, </ins> }, }

is always using the cross namespace context. This needs to be fixed to support scoped watches.


Question: []string{"runtime_handler"}, ) <del> // RunningPodCount is a gauge that tracks the number of Pods currently running </del> <ins> // RunningPodCount is a gauge that tracks the number of Pods currently with a running sandbox // It is used to expose the kubelet internal state: how many pods have running containers in the container runtime, and mainly for debugging purpose. </ins> RunningPodCount = metrics.NewGauge( &metrics.GaugeOpts{ Subsystem:      KubeletSubsystem, Name:           RunningPodsKey, <del> Help:           "Number of pods currently running", </del> <ins> Help:           "Number of pods that have a running pod sandbox", </ins> StabilityLevel: metrics.ALPHA, }, )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet metric should only count pods in state. But it incorrectly reports the sum of pods in + state. Only pods in state should be counted. Create a pod with sample yaml to create a pod that stays in state. Verify that pod we created is in state, Now let's count the total number of pods using Out of 15, 1 is in and rest are in state. But the corresponding metric from Prometheus dashboard returns 15. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Does it mean that pod after initial will be take into account for running pod metrics ? According the code, it counts sandboxes. Once Pod pause container is started, and it will be counted.
/triage accepted
As some comments in , the have a different meanings between and .
Yeah. But then that means metric ends up misleading the user. Would it be possible to use API Pod instead of runtime pod?
I may follow the rules in to make it consistent with
Please feel free to tag me for review once you submit the PR.
I am wondering if we should just update the documentation for stating explicitly that the results may not always match with the due to differences between them in what they consider as a pod. What do you think?
Agree. Generally, metrics should be simple to calculate. The current logic makes sense from the runtime side. Proposal 1: update the doc for . Proposal 2: rename it to something like or an accuracy name. Proposal 3: make it the same meaning as (which need some further calculation)
I think I will go with Proposal 1. The ideal thing would be Proposal 3, but I am afraid we might be underestimating the effort required. What do you think?
IIRC, is not intended to reflect the pod phase in the API server. It is used to expose the kubelet internal state: how many pods have running containers in the container runtime, and mainly for debugging purpose.

To make it more clear: to A pod is in phase when kubelet tries to create init container after pause container started.
Thanks Closing this because we have more clarity on what that metric is supposed to mean.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry closed it before noticing Reopened this bug.


Question: name = "all-srcs", srcs = [ ":package-srcs", <ins> "//staging/src/k8s.io/apiserver/plugin/pkg/audit/buffered:all-srcs", </ins> "//staging/src/k8s.io/apiserver/plugin/pkg/audit/log:all-srcs", "//staging/src/k8s.io/apiserver/plugin/pkg/audit/webhook:all-srcs", ],

We should introduce a buffer (similar to what we have in the webhook output backend) to the log file output backend. Audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. If an audit event is output when the channel is full, it should be dropped (this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold). This behavior should be optional (configurable) /cc is tracking the regression
Naturally os will cache , these cached data will be written to disk either by fsync or pdflush periodically. What do you think is the most time-costing procedure in audit?
The problem is not with the time consumed. The problem is that currently write is performed synchronously with processing the request, which slows the request down, no matter how well caching works
The problem is that currently write is performed synchronously with processing the request Sure synchronously write does cost time serializing. Introduce buffer mechanism can really reduce api request latency time.
It doesn't matter that much if it happens asynchronously Indeed
Yes, this is a good point to improve performance.
Now I totally comprehend this. I am pleased to implement this buffer.
I'll be happy to review that
audit log is important sensitive information which we should try to make sure it reaches disk at all cost. Does this effort compromises that in favor of optimizing for latency ?
I'll make sure it's configurable and if the security is critical, you can setup synchronous write in your deployment. Having said that, it's critical to be able to accommodate the case where having 100% calls written does not compensate for increased latencies. Does it make sense?
thanks my preference would be to keep the default as no buffering when making it configurable. I didnt understand how does one setup "synchronous write in your deployment" and how that will affect audit logs reaching disk. My worry is mainly around a crash where the buffered audit logs dont reach the disk and are lost
if use buffer, audit events lost is inevitable when crash. As like webhook backend, still exist this risk. But to improve apiserver qps, something should comprise. By the way, we still reserve the synchronous write mode, which is configurable.

To make it simple, when offered trade-off, people are more likely to give up audit logs reliability than performance. Makes sense?
i am not against this change. i think what would be best is to make the default as no buffering and enable buffering through a parameter. Also for Enterprises, the trade off is not correct, they would prefer to NOT give up audit logs
I understand what you're suggesting and I don't agree :) Switching to buffering will for sure increase the latency, but will only increase the probability of losing logs, which will never be zero. I'm pretty sure we need to have buffered mode by default. However, since this argument is about making a trade-off and since neither of us represents quorum, let's ask more people for their opinion What do you think?
The default is slightly less important here since audit logging as a whole is not enabled by default. If we do turn it on by default, it becomes more important. That said, my hunch is that we should default to the mode with lower performance impact, but I don't have strong feelings either way.
As long as there are options to turn it on of off (the synchronous write) I'm fine with either default. Still, I'd like to see due diligence towards ensuring low lost rate when using buffered write, and sufficient documentation around caveats of using one or the other. Audit, from my personal experience, is crucial for majority of enterprises and they won't tolerate not even single lost event.
That is not possible, there will always be a situation when an audit event can be lost: sudden power loss, disk/network failure, random bit flip, etc. The questions is whether 10^(-6) is low enough loss probability, or we need to get 10^(-14) by sacrificing performance (arbitrary numbers, just to illustrate my point). Note, that reduced performance can increase the number of timeouts and retries, increasing load and leading to failures and even more lost events.
I think this is an important bug that we're not testing audit logging at scale
Marking it as urgent, since it's blocking
If split into 3 prs, I must do it one by one . Now first one is up .
SGTM Status of this issue: ETA: 1/12/2017 Risks: Complicated fix required
has been reviewed for almost 2 months and has not done.

If I raise 3 prs to replace it, I am still worried whether all of them can be merged. They act as a whole.
Now I'm going to devote my time to it, we'll do it
OK.
In progress, reviewed first PR, waiting for applying changes ETA: Dec 01
Per Exception discussion, this portion of the feature is being removed from the milestone. As such, changing the priority. /priority important-soon /remove-priority critical-urgent
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: //count the evictionsNumber zone := nodetopology.GetZoneKey(node) evictionsNumber.WithLabelValues(zone).Inc() <ins> evictionsTotal.WithLabelValues(zone).Inc() </ins> } return result, 0

I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted


Question: func TestUnschedulablePodBecomesSchedulable(t *testing.T) { tests := []struct { <del> name   string init   func(kubernetes.Interface, string) error pod    *testutils.PausePodConfig update func(kubernetes.Interface, string) error </del> <ins> name                   string init                   func(kubernetes.Interface, string) error pod                    *testutils.PausePodConfig update                 func(kubernetes.Interface, string) error enableReadWriteOncePod bool </ins> }{ { name: "node gets added",

We need an integration test for the scheduler enforcement of the alpha feature. This is required for beta graduation; we need to collect signal on the current behavior before we can proceed. See this comment for context:
/sig storage /sig scheduler
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig storage /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: return "", err } <ins> func CreatePDWithRetry() (string, error) { return createPDWithRetry("") } func CreatePDWithRetryAndZone(zone string) (string, error) { return createPDWithRetry(zone) } </ins> func DeletePDWithRetry(diskName string) error { var err error for start := time.Now(); time.Since(start) < PDRetryTimeout; time.Sleep(PDRetryPollTime) {

For statically created PVs, the persistent volume label admission controller adds the zone label to the PV object. If the admission controller is not enabled or has some bugs, then an e2e test would be helpful for catching these errors. /assign
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Question: func (pip *parallelImagePuller) pullImage(ctx context.Context, spec kubecontainer.ImageSpec, pullSecrets []v1.Secret, pullChan chan<- pullResult, podSandboxConfig *runtimeapi.PodSandboxConfig) { go func() { <ins> startTime := time.Now() </ins> imageRef, err := pip.imageService.PullImage(ctx, spec, pullSecrets, podSandboxConfig) pullChan <- pullResult{ <del> imageRef: imageRef, err:      err, </del> <ins> imageRef:     imageRef, err:          err, pullDuration: time.Since(startTime), </ins> } }() }

While going over Kubernetes events I noticed that image pulls for some small images were taking several minutes according to the "Successfully pulled image" event. From investigating this is misleading as the actual pull time is much smaller but due to --serialize-image-pulls=true being the default in the observed cluster and other pods being initialized at the time, the reported pull time ends up accounting for time while waiting for other pods to finish pulling. I expected to see the time it takes to download the image itself from a registry. Customers seem to rely on this metric for this information and we have received some complaints as a result where customers noted slow pull times as a result of misunderstanding the current behavior of this metric. Start with a K8s cluster with --serialize-image-pulls=true (default) - For my check I used default minikube v1.26.1 pod-large where pod-large pulls any large image (the image cannot be locally cached) after step 1 (within seconds) create pod-small where pod-small pulls any small image Check the output for the event after running: Note that despite the small size, pull time for the image in pod-small is several minutes, we should instead see it take seconds (which is how long the pull actually takes). To note this is the case you can try pulling another small image like alpine that isnt cached locally and compare after the above are done. Here is the code that pulls an image in kubernetes and logs the event: As can be seen from it, it's not true pull time, just time from invoking the puller to receiving a result. The serial puller is using a Golang channel as a queue and will just wait until a previous image is done pulling as I noted before, thus it will end up recording the amount of time in the queue + pull time yielding a misleading metric. <details</details<detailsShould reproduce anywhere (tried it on minikube) but have personally observed this in Azure as well </details<details</details<details</details<details</details<details</details
/sig node
But means that kubelet can pull multi images at the same time. For kubelet, is by default true. is for promoting the use of parallel image pulls. The problem maybe with . I did some simple test with 3 ubuntu images as below. The time was collected from events.

-- --serialize-image-pulls=true -- -- image pulling serialize ubuntu:22.10 73.6 MiB 1m21.284220239s ubuntu:20.04 75.7 MiB 2m7.876426944s ubuntu:14.04 212.9 MiB 4m52.884660938s
You are right thanks for taking a look at this. It seems I got the two reversed in my head. The issue indeed happens when --serialize-image-pulls=true which is the default as you pointed out. I'll update the issue to reflect that
/triage accepted /priority important-longterm


Question: <ins> # Replication Controller ## What is a _replication controller_? A _replication controller_ ensures that a specified number of pod "replicas" are running at any one time.  If there are too many, it will kill some.  If there are too few, it will start more. As opposed to just creating singleton pods or even creating pods in bulk, a replication controller replaces pods that are deleted or terminated for any reason, such as in the case of node failure. For this reason, we recommend that you use a replication controller even if your application requires only a single pod. As discussed in [life of a pod](pod-states.md), `replicationController` is *only* appropriate for pods with `RestartPolicy = Always`.  `ReplicationController` should refuse to instantiate any pod that has a different restart policy. As discussed in [issue #503](https://github.com/GoogleCloudPlatform/kubernetes/issues/503#issuecomment-50169443), we expect other types of controllers to be added to Kubernetes to handle other types of workloads, such as build/test and batch workloads, in the future. A replication controller will never terminate on its own, but it isn't expected to be as long-lived as services. Services may be comprised of pods controlled by multiple replication controllers, and it is expected that many replication controllers may be created and destroyed over the lifetime of a service. Both services themselves and their clients should remain oblivious to the replication controllers that maintain the pods of the services. ## How does a replication controller work? ### Pod template A replication controller creates new pods from a template, which is currently inline in the `replicationController` object, but which we plan to extract into its own resource [#170](https://github.com/GoogleCloudPlatform/kubernetes/issues/170). Rather than specifying the current desired state of all replicas, pod templates are like cookie cutters. Once a cookie has been cut, the cookie has no relationship to the cutter. There is no quantum entanglement. Subsequent changes to the template or even switching to a new template has no direct effect on the pods already created. Similarly, pods created by a replication controller may subsequently be updated directly. This is in deliberate contrast to pods, which do specify the current desired state of all containers belonging to the pod. This approach radically simplifies system semantics and increases the flexibility of the primitive, as demonstrated by the use cases explained below. Pods created by a replication controller are intended to be fungible and semantically identical, though their configurations may become heterogeneous over time. This is an obvious fit for replicated stateless servers, but replication controllers can also be used to maintain availability of master-elected, sharded, and worker-pool applications. Such applications should use dynamic work assignment mechanisms, such as the [etcd lock module](https://coreos.com/docs/distributed-configuration/etcd-modules/) or [RabbitMQ work queues](https://www.rabbitmq.com/tutorials/tutorial-two-python.html), as opposed to static/one-time customization of the configuration of each pod, which is considerined an anti-pattern. Any pod customization performed, such as vertical auto-sizing of resources (e.g., cpu or memory), should be performed by another online controller process, not unlike the replication controller itself. ### Labels The population of pods that a `replicationController` is monitoring is defined with a [label selector](labels.md), which creates a loosely coupled relationship between the controller and the pods controlled, in contrast to pods, which are more tightly coupled. We deliberately chose not to represent the set of pods controlled using a fixed-length array of pod specifications, because our experience is that that approach increases complexity of management operations, for both clients and the system. The replication controller should verify that the pods created from the specified template have labels that match its label selector. Though it isn't verified yet, you should also ensure that only one replication controller controls any given pod, by ensuring that the label selectors of replication controllers do not target overlapping sets. Note that `replicationControllers` may themselves have labels and would generally carry the labels their corresponding pods have in common, but these labels do not affect the behavior of the replication controllers. Pods may be removed from a replication controller's target set by changing their labels. This technique may be used to remove pods from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically (assuming that the number of replicas is not also changed). Similarly, deleting a replication controller does not affect the pods it created. It's `replicas` field must first be set to 0 in order to delete the pods controlled. In the future, we may provide a feature to do this and the deletion in a single client operation. ## Responsibilities of the replication controller The replication controller simply ensures that the desired number of pods matches its label selector and are operational. Currently, only terminated pods are excluded from its count. In the future, [readiness](https://github.com/GoogleCloudPlatform/kubernetes/issues/620) and other information available from the system may be taken into account, we may add more controls over the replacement policy, and we plan to emit events that could be used by external clients to implement arbitrarily sophisticated replacement and/or scale-down policies. The replication controller is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://github.com/GoogleCloudPlatform/kubernetes/issues/492)), which would change its `replicas` field. We will not add scheduling policies (e.g., [spreading](https://github.com/GoogleCloudPlatform/kubernetes/issues/367#issuecomment-48428019)) to replication controller. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsehwere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://github.com/GoogleCloudPlatform/kubernetes/issues/170)). The replication controller is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubecfg (run, stop, resize, rollingupdate) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing replication controllers, auto-scalers, services, scheduling policies, canaries, etc. ## Common usage patterns ### Rescheduling As mentioned above, whether you have 1 pod you want to keep running, or 1000, replication controller will ensure that the specified number of pods exists, even in the event of node failure or pod termination (e.g., due to an action by another control agent). ### Scaling Replication controller makes it easy to scale the number of replicas up or down, either manually or by an auto-scaling control agent, by simply updating the `replicas` field. ### Rolling updates Replication controller is designed to facilitate rolling updates to a service by replacing pods one-by-one. As explained in [#1353](https://github.com/GoogleCloudPlatform/kubernetes/issues/1353), the recommended approach is to create a new replication controller with 1 replica, resize the new (+1) and old (-1) controllers one by one, and then delete the old controller after it reaches 0 replicas. This predictably updates the set of pods regardless of unexpected failures. Ideally, the rolling update controller would take application readiness into account, and would ensure that a sufficient number of pods were productively serving at any given time. The two replication controllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates. ### Multiple release tracks In addition to running multiple releases of an application while a rolling update is in progress, it's common to run multiple releases for an extended period of time, or even continuously, using multiple release tracks. The tracks would be differentiated by labels. For instance, a service might target all pods with `tier in (frontend), environment in (prod)`.  Now say you have 10 replicated pods that make up this tier.  But you want to be able to 'canary' a new version of this component.  You could set up a `replicationController` with `replicas` set to 9 for the bulk of the replicas, with labels `tier=frontend, environment=prod, track=stable`, and another `replicationController` with `replicas` set to 1 for the canary, with labels `tier=frontend, environment=prod, track=canary`.  Now the service is covering both the canary and non-canary pods.  But you can mess with the `replicationControllers` separately to test things out, monitor the results, etc. </ins>

Currently replication controllers define an exact value. Are there disadvantages to allowing the replication controller to define more of a policy - at least X, at most X, exactly X? Since replication controllers aren't really aware of overlapping sets, I guess you can't really use that to guide allocation between two pools because they would just end up fighting unproductively
What is the use case? This seems sort of like auto-scaling policy. I'd prefer to put those objects in the auto-scaler.
It sounds like you want a canary-controller that scales the old version down while scaling the new one up. Combine with a service that spreads traffic across both versions and this should Just Work (tm).
I'm thinking about this from the mechanics of doing a deployment by composing replication controllers. With controllers as they are today: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo,deployment=2 x0 alter controller 2 to be x1 (or up by the batch size) alter controller 1 to be x4 (or down by the batch size) ... repeat until done delete controller 1 The thought around atLeast was about explicitly representing the invariant you want during an available deployment: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo, atLeast=5 set controller to be atMost=5 delete a pod from the name=foo,deployment=1 query ... continue until 0 pods in that query delete controller 1 update controller 2 to change the label query to name=foo,deployment=2 I'm not sure that's better.... just the for loop for deletes seems more reasonable. Not a huge deal, just trying to work through how we'd model common deploy cases for end users.
Overlapping replicationController sets: I think we should validate that this isn't possible upon creation of a replicationController. We can do this by determining whether the label selector may overlap with any others, though to do this analytically without looking at the combinations of labels attached to pods it requires that all replicationController label selectors have at least one key in common, with a different value than all other replicationControllers. Policy: As mentioned, we intended the policy to go in a separate auto-scaling policy entity, in a higher-level layer / separate component.

Deployment: As pointed out, it is possible to create a new replicationController and gradually scale it up while scaling the original down. Or, one could do the Asgard trick of creating a new replicationController and then shift traffic to it. It's possible to add/remove instances from a service by tweaking their labels. For instance one could toggle the value of an in_service=true/false label. The update demo changes the template of a single replication controller and then kills instances one by one until they all get replaced. I don't like that due to the risk of unpredictable replacements and complexity involved with rollbacks. Splitting out the template ( ) would address the rollback issue, but not the unpredictability. It seems to me that the atMost approach suffers from the same risk. That said, I see what you mean that atLeast/atMost are not auto-scaling policies, but are invariants to maintain. I could imagine that flexibility could be useful. For example, a worker pool manager could use that to opportunistically request resources, where some pod requests may fail. Or, perhaps in the case that we need to temporarily create a new pod for some reason (CRIU-based migration?) but we don't want an old one to be killed. So, I like the idea, just not the use case. :-) FWIW, exactly X could be implemented by combining atLeast and atMost.
Agree that without a kill policy on the replication controller the uses cases described don't work. Is a kill policy something you've discussed for the replication controller (oldest, ones that don't match my template first, ones on most overloaded systems)?
I agree that kill policy would be useful and all of the policies you mention are reasonable in particular scenarios. However, I see this as a rich, unbounded policy space, much like scheduling. IIRC, Joe and/or Brendan proposed kill policies early on and I asked for them to be removed. Instead, I propose that we leave the default policy entirely up to the system, much like scheduling, so we can take as many factors into account as we like, and otherwise I'd like to enable the user to implement an arbitrary policy using event hooks . For instance, I could imagine POSTing to a URL whenever a pod needed to be killed.

The user's hook would have N seconds to respond (perhaps configurable), after which the replication controller would kill its default choice.
Thanks for the clarification - I think I'll close this for now until the API stabilizes, and then add some of this context to the doc.
I suggest re-opening this issue, but change the title to "add XXX context to doc YYY", so that we can reference in the PR for posterity.


Question: // Returns selectors of services, RCs and RSs matching the given pod. func getSelectors(pod *v1.Pod, sl algorithm.ServiceLister, cl algorithm.ControllerLister, rsl algorithm.ReplicaSetLister, ssl algorithm.StatefulSetLister) []labels.Selector { <del> selectors := make([]labels.Selector, 0, 3) </del> <ins> var selectors []labels.Selector </ins> if services, err := sl.GetPodServices(pod); err == nil { for _, service := range services { selectors = append(selectors, labels.SelectorFromSet(service.Spec.Selector))

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.


Question: "time" v1 "k8s.io/api/core/v1" <del> "k8s.io/apimachinery/pkg/api/resource" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/util/uuid" runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" "k8s.io/klog/v2" </del> kubeletpodresourcesv1 "k8s.io/kubelet/pkg/apis/podresources/v1" "k8s.io/kubernetes/pkg/kubelet/apis/podresources" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager"

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: expectNoEvent(watchB, watch.Modified, testPodA) testPodA, err = updatePod(f, testPodA.GetName(), func(p *v1.Pod) { <del> p.Spec.ActiveDeadlineSeconds = int64ptr(5) </del> <ins> p.ObjectMeta.Labels["mutation"] = "2" </ins> }) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Modified, testPodA)

https://k8s- Please work on finding a fix for the flake. /priority failing-test /priority important-soon /sig api-machinery /kind flake
/assign
I think this was fixed by
we had a failure on 3/28 (after the above fix merged). https://k8s- Looking at the logs this seems to be a different error than the one fixed by . Can you confirm plz?
Hmm that is different, and I will look into the reason.
There hasn't been a failure since 3/28, So while the test might have flakes it is definitely less flaky than 12 failures per week now
Yes the latest runs look good. Please close the issue
/close


Question: Output: true, }, { <ins> Inspected: dockertypes.ImageInspect{ ID:       "sha256:9bbdf247c91345f0789c10f50a57e36a667af1189687ad1de88a6243d05a2227", RepoTags: []string{"docker.io/busybox:latest"}, }, Image:  "docker.io/library/busybox:latest", Output: true, }, { </ins> // RepoDigest match is is required Inspected: dockertypes.ImageInspect{ ID:          "",

bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it.

So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it.

Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?


Question: string ip = 1; } // PodSandboxNetworkStatus is the status of the network for a PodSandbox. <ins> // Currently ignored for pods sharing the host networking namespace. </ins> message PodSandboxNetworkStatus { // IP address of the PodSandbox. string ip = 1;

What happened: We have installed k8s cluster on openstack cloud provider setup. K8s cluster is dual-stack enabled. During fresh deployment or worker node reboots (master nodes , worker nodes) , k8s control plane pods (other host networking pods aswell) such as kube-proxy, kube-scheduler , kube-controller-manager get only 1 IP address instead of dual-stack address. What you expected to happen: We expect all host networking pods to get dual-stack addresses , since cluster is dual-stack enabled. How to reproduce it (as minimally and precisely as possible): a) Openstack cloud provider setup with minimum 4-5 VMs. b) 3 master nodes, 2 worker nodes c) Execute simultaneous reboot of all nodes. Environment: Kubernetes version : v1.22.0
/sig node
/sig network /triage needs-information /assign Hi When you describe your node what are the IPs assigned in node object? kubelet copies from node object into pods. The way they are set on node object is by the cloud provider, manual via kubelet's flag or auto-detect which prefers IPv4 (check: ).
when i describe the node object , they contain dual-stack addresses. Yes i debugged kubelet with few additional logs. I could see that when kubelet is reading node obj to create these static pods, it sees only 1 IP. But eventually node object is populated with both the addresses. I also understand, is for baremetal installation. But we have cloud provider setup which might be delaying setting both the IPs in node obj. Before cloud provider sets both the IPs ,kubelet is fast enough to create these static pods. Let me know if you need any additional logs.
thanks again Few additional questions: are these static pods? When the node object gets its dualstack addresses and pods were created after the node object updates do you still get single address pods?
I've chatted with the reporter in slack, it seems a race on populating the node addresses by the cloud-provider,
Yes. It would seem so. Two things here i am trying to clarify. cloud providers node register with taints (no schedule) until cloud provider adds provider-Id and other details including IPs.

these are static pods then I am not sure if kubelet respects no-schedule (those are respected by scheduler which is out of the picture for static pods) If they are then they will run - most likely - before node gets IP addresses from cloud provider. The questions i asked above should put some lights on this.
Yes i am referring to kube-proxy , kube-api-server ,kube-scheduler pods which use host-networking. This is not under our control. I mean , we are just rebooting the nodes in quick succession and at the end we just check the IP addresses of the above static pods. As i see the code , kubelet brings them up as soon as he is UP.
IMO the cloud provider really needs to figure out the IPs before kubelet starts. OR the node startup needs to block until both IP addresses are available so that the cloud provider and/or kubelet can read them.
Should we accept this as a bug? Do we think we need to force some taint on new nodes that cloud-providers need to clear (obviously not compatible, but you get the idea)?
-- There is a no-schedule taint already there (if cloud provider is configured on kubelet). It is a bug (or undocumented behavior). I think we should always update PodIPs in host network mode. /triage accepted
xref
so i looked at this again. I believe should fix this issue as well. i don't think we should place this behind a feature gate. We already update when the host IP changes. We just need to edit the release note of to describe the behavior and caveats. Thoughts?
Please ignore that comment. The PR does not solve for
/remove-sig node
: Any more info needed from me ?
no thank you. We are addressing this by and the related KEP.
, any idea when will the fix be available ?


Question: checkedICC: true, result:     "test", <del> ok:         false, </del> <ins> overridden: false, }, "overridden context used to verify explicit namespace in config": { clientConfig: &testClientConfig{ namespace:          "default", namespaceSpecified: false, // a namespace that comes from a context is not considered overridden rawconfig:          &clientcmdapi.Config{Contexts: map[string]*clientcmdapi.Context{"overridden-context": {Namespace: "default"}}}, }, overrides: &ConfigOverrides{CurrentContext: "overridden-context"}, icc: &testICC{ possible: true, testClientConfig: testClientConfig{ namespace:          "icc", namespaceSpecified: false, // a namespace that comes from icc is not considered overridden }, }, checkedICC: true, result:     "default", overridden: false, // a namespace that comes from a context is not considered overridden </ins> }, } for name, test := range testCases { <del> c := &DeferredLoadingClientConfig{icc: test.icc} c.clientConfig = test.clientConfig ns, ok, err := c.Namespace() if test.icc.called != test.checkedICC { t.Errorf("%s: unexpected in-cluster-config call %t", name, test.icc.called) } if err != test.err || ns != test.result || ok != test.ok { t.Errorf("%s: unexpected result: %v %s %t", name, err, ns, ok) } </del> <ins> t.Run(name, func(t *testing.T) { c := &DeferredLoadingClientConfig{icc: test.icc, overrides: test.overrides} c.clientConfig = test.clientConfig ns, overridden, err := c.Namespace() if test.icc.called != test.checkedICC { t.Errorf("%s: unexpected in-cluster-config call %t", name, test.icc.called) } if err != test.err || ns != test.result || overridden != test.overridden { t.Errorf("%s: unexpected result: %v %s %t", name, err, ns, overridden) } }) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In-cluster namespace isn't overridden by kubeconfig if the context is specified via kubectl flag. When using , the override works as expected. What you expected to happen: If I specify for a context in kubeconfig, then I expect to use as the default namespace, regardless of whatever's in . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: supposedly fixed this issue, but not for the case where the context is specified via a command line flag (). This is because this line checks for . Environment: Kubernetes version (use ): v1.10.5 (details above) Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): - Kernel (e.g. ): - Install tools: - Others: - <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in


Question: errors := []error{} if s.BindPort < 0 || s.BindPort > 65535 { <del> errors = append(errors, fmt.Errorf("--secure-port %v must be between 0 and 65535, inclusive. 0 for turning off secure port.", s.BindPort)) </del> <ins> errors = append(errors, fmt.Errorf("--secure-port %v must be between 0 and 65535, inclusive. 0 for turning off secure port", s.BindPort)) </ins> } return errors

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: return nil } <del> hyphenIndex := strings.Index(value, "-") if hyphenIndex == -1 { return fmt.Errorf("expected hyphen in port range") </del> <ins> var err error var low, high int var notation int if plusIndex == -1 && hyphenIndex == -1 { notation |= SinglePortNotation } if hyphenIndex != -1 { notation |= HyphenNotation } if plusIndex != -1 { notation |= PlusNotation </ins> } <del> var err error var low int var high int low, err = strconv.Atoi(value[:hyphenIndex]) if err == nil { </del> <ins> switch notation { case SinglePortNotation: var port int port, err = strconv.Atoi(value) if err != nil { return err } low = port high = port case HyphenNotation: low, err = strconv.Atoi(value[:hyphenIndex]) if err != nil { return err } </ins> high, err = strconv.Atoi(value[hyphenIndex+1:]) <del> } if err != nil { return fmt.Errorf("unable to parse port range: %s: %v", value, err) </del> <ins> if err != nil { return err } case PlusNotation: var offset int low, err = strconv.Atoi(value[:plusIndex]) if err != nil { return err } offset, err = strconv.Atoi(value[plusIndex+1:]) if err != nil { return err } high = low + offset default: return fmt.Errorf("unable to parse port range: %s", value) </ins> } if low > 65535 || high > 65535 {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Implement this TODO: What you expected to happen: Start kube-proxy with a 'beginPort+offset' format port range configuration. This is needed when I want to allocate specific numbers of port from a host without extra manual calculation. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/sig network Does anyone besides kube-proxy use this?
/assign


Question: } } <ins> func TestTranslateInTreeInlineVolumeToCSINameUniqueness(t *testing.T) { for driverName := range inTreePlugins { t.Run(driverName, func(t *testing.T) { ctl := New() vs1, err := generateUniqueVolumeSource(driverName) if err != nil { t.Fatalf("Couldn't generate random source: %v", err) } pv1, err := ctl.TranslateInTreeInlineVolumeToCSI(&v1.Volume{ VolumeSource: vs1, }) if err != nil { t.Fatalf("Error when translating to CSI: %v", err) } vs2, err := generateUniqueVolumeSource(driverName) if err != nil { t.Fatalf("Couldn't generate random source: %v", err) } pv2, err := ctl.TranslateInTreeInlineVolumeToCSI(&v1.Volume{ VolumeSource: vs2, }) if err != nil { t.Fatalf("Error when translating to CSI: %v", err) } if pv1 == nil || pv2 == nil { t.Fatalf("Did not expect either pv1: %v or pv2: %v to be nil", pv1, pv2) } if pv1.Name == pv2.Name { t.Errorf("PV name %s not sufficiently unique for different volumes", pv1.Name) } }) } } func generateUniqueVolumeSource(driverName string) (v1.VolumeSource, error) { switch driverName { case plugins.GCEPDDriverName: return v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: string(uuid.NewUUID()), }, }, nil case plugins.AWSEBSDriverName: return v1.VolumeSource{ AWSElasticBlockStore: &v1.AWSElasticBlockStoreVolumeSource{ VolumeID: string(uuid.NewUUID()), }, }, nil case plugins.CinderDriverName: return v1.VolumeSource{ Cinder: &v1.CinderVolumeSource{ VolumeID: string(uuid.NewUUID()), }, }, nil case plugins.AzureDiskDriverName: return v1.VolumeSource{ AzureDisk: &v1.AzureDiskVolumeSource{ DiskName:    string(uuid.NewUUID()), DataDiskURI: string(uuid.NewUUID()), }, }, nil case plugins.AzureFileDriverName: return v1.VolumeSource{ AzureFile: &v1.AzureFileVolumeSource{ SecretName: string(uuid.NewUUID()), ShareName:  string(uuid.NewUUID()), }, }, nil default: return v1.VolumeSource{}, fmt.Errorf("couldn't find logic for driver: %v", driverName) } } </ins> func TestPluginNameMappings(t *testing.T) { testCases := []struct { name             string

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: type timeBudgetImpl struct { sync.Mutex <del> budget time.Duration refresh   time.Duration </del> <ins> clock     clock.Clock budget    time.Duration </ins> maxBudget time.Duration <ins> refresh   time.Duration // last store last access time last time.Time </ins> } <del> func newTimeBudget(stopCh <-chan struct{}) timeBudget { </del> <ins> func newTimeBudget() timeBudget { </ins> result := &timeBudgetImpl{ <ins> clock:     clock.RealClock{}, </ins> budget:    time.Duration(0), refresh:   refreshPerSecond, maxBudget: maxBudget, } <del> go result.periodicallyRefresh(stopCh) </del> <ins> result.last = result.clock.Now() </ins> return result } <del> func (t *timeBudgetImpl) periodicallyRefresh(stopCh <-chan struct{}) { ticker := time.NewTicker(time.Second) defer ticker.Stop() for { select { case <-ticker.C: t.Lock() if t.budget = t.budget + t.refresh; t.budget > t.maxBudget { t.budget = t.maxBudget } t.Unlock() case <-stopCh: return } } } </del> func (t *timeBudgetImpl) takeAvailable() time.Duration { t.Lock() defer t.Unlock() <ins> // budget accumulated since last access now := t.clock.Now() acc := now.Sub(t.last).Seconds() * t.refresh.Seconds() if acc < 0 { acc = 0 } // update current budget and store the current time if t.budget = t.budget + time.Duration(acc*1e9); t.budget > t.maxBudget { t.budget = t.maxBudget } t.last = now </ins> result := t.budget t.budget = time.Duration(0) return result

Instead of relying on the clock: we should do something similar to the golang rate-limitter is doing:
/sig api-machinery /priority important-longterm
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to help on this.
/assign Thanks! /triage accepted
can you please provide more context or some guidance on how to proceed?
Sorry for delayed response. - what we should do is to basically: instead of having an additional go-routine that every 1s refreshes the budget change it roughly to: (a) keep the value of budget and the last time when we were refreshing it (b) on takeAvailable method: first recompute takeAvailable, which will be "min(current budget + time elapsed since last refresh * refreshValue, maxAllowedBudget) update last refresh time to now return that value [We also would need to update the existing tests to match the new logic.] - would you be willing to work on it?
-- I can take this up. One question I had was should the method rely on or the current time should be passed on by the caller like in the following: Having the caller pass the time completely removes the dependency on the clock at least for the implementation.
/assign
Let's may be start with: passing on budget creation to it relying on
-- thank you for the thoughts. I drafted a rough implementation in which still depends on . I will make the change as you suggested and rely solely on a instance passed on initialization. I have to rethink the testing strategy a bit, possibly using a .
Yes - FakeClock is the way to test this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopening - this didn't happen.


Question: } fs.IPVar(&s.BindAddress, "address", s.BindAddress, <del> "The IP address on which to serve the insecure --port (set to 0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).") </del> <ins> "The IP address on which to serve the insecure --port (set to '0.0.0.0' or '::' for listening in all interfaces and IP families).") </ins> fs.MarkDeprecated("address", "see --bind-address instead.") fs.Lookup("address").Hidden = false

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are flag descriptions where is stated that the unspecified address behaves differently depending on the IP family: However, that's not the behavior implemented in and , clarified in the golang documentation What you expected to happen: Fix the comment to state the reality, i.e How to reproduce it (as minimally and precisely as possible): From the root of the kubernetes repo Anything else we need to know?: xref Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind cleanup /help wanted /good-first-issue /priority important-longterm /cc
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign /remove-help
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
For the flag in kube-proxy, is used if the address is not specified and can't be resolved by hostname, so I suggest to remove this comment: 0.0.0.0::` for all IPv6 interfaces'
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I am very much in favor of this. Being correct and precise in these flags is valuable. To convince myself you were right, I wrote this program: And then I ran it for various inputs: In these flags and docs, we should be clear that if you are specifying a v6 addr and port you have to use syntax. Or maybe we should just audit and fix ALL such places to support and document that "nothing" is how you specify all interfaces. E.g. --bind-address=":80" is more obvious than either "0.0.0.0:80" or "[::]:80".

Even if those syntaxes work, let's just make "" the norm?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: package network import ( <ins> "bytes" </ins> "context" <ins> "encoding/json" </ins> "fmt" "math" "net/http"

[x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance Where/How does PATCH/HEAD/OPTIONS requests for the following endpoints line up with the API reference? connectCoreV1PatchNamespacedPodProxyWithPath connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath Pod v1 core: Proxy Operations the following endpoints are listed as valid endpoints; connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath These requests are getting logged as a verb. Line : api/openapi-spec Line : api/openapi-spec Looking at there doesn't look to be a mapping for the above methods. According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Create a service and link it to the pod Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/re-open
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: clientset "k8s.io/client-go/kubernetes" clientsetscheme "k8s.io/client-go/kubernetes/scheme" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" <ins> kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants" </ins> kubeadmutil "k8s.io/kubernetes/cmd/kubeadm/app/util" "k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient" kubeproxyconfigscheme "k8s.io/kubernetes/pkg/proxy/apis/kubeproxyconfig/scheme"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed.

Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?

if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well