Question: "k8s.io/kubernetes/pkg/apis/scheduling" kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/test/e2e/framework" <del> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </del> imageutils "k8s.io/kubernetes/test/utils/image" "github.com/onsi/ginkgo"

It is unclear why the test needs the dynamic kubelet config. Need to clean it up. /sig node /priority important-soon /help Test is not a part of any testgrid. Need to understand why and return it back to the grid. Also it is marked as flaky. Need to be deflaked. unknown No response No response No response_ /sig node
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### Which jobs are failing? /priority important-soon /help ### Which tests are failing? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I am new to this project and would really love to start contributing and thought this might be a good start. Could you tell me a little more about this problem?
glad to have you, welcome! Some step by step you may start with: Find the test mentioned, understand what it is doing If dependency on DynamicKubeletConfig is found, try to work around them and remove that line. Try running the test locally. If you can repro flakiness - try to fix it If test passes: Try removing the and see if passes in serial tests on presubmits
Thank you so much!

/assign
/cc
it had a non-op config change in the past - - I potentially missed this when removing that
so, if I'd like to fix it, what should I do?
see Since we understood item 2 - it is not needed, just remove the skipper lines. Ideally test needs to be de-flaked as well, but it it not necessarily in the scope of this change
this is a simple issue, so I tried to fix it since I've found it some days.


Question: padding: " ", want:    " a ", }, <ins> { noun:    "S", padding: " ", want:    " a ", }, { noun:    "O", padding: " ", want:    " an ", }, </ins> } for _, tt := range tests { if got := GetArticleForNoun(tt.noun, tt.padding); got != tt.want {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add a CRD resource with kind only a single character while cause kube-apiserver crashslice bounds out of range What you expected to happen: add validation to limit CRD kind definition or fix the bug of slice out of bounds How to reproduce it (as minimally and precisely as possible): apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: version: v1 names: kind: R plural: rs singular: r scope: Cluster Anything else we need to know?: apiserverlogs: 10:05:32. 6 ] Shutting down OpenAPI controller E6606 10:05:32. 6 ] Observed a panic: runtime.boundsError{x:-1, y:1, signed:true, code:0x3(runtime error: slice bounds out of range [-1:]) goroutine 1670 [running]: (0x41458a0, Gxc00ed66d40) /workspace/anago-v1.17.G- +0xa3 (0x0, 0x0, 0x0) /workspace/anago-v1.17.G- +0x82 panic(0x41458a0, Oxc0Oed66d40) +0x1b2 (0x7493232, 0x1, 0x447d778, 0x1, 0x130, 0xc00ed13680) /workspace/anago-v1.17.0- +0x2b8 ("builder).descriptionFor(€xc007b351f0, 0x0, 0x0, 0x4481982, 0x6, Gxc00ed06d00, 0x18) /workspace/anago-v1.17.0- +0x3c6 ("builder).buildRoute(GxcGG7b351fG, 0xc00ed66d00, 0x18, 0x0, 0x0, 0x447f0F3, 0x4, 0x447f71b, 0x4,

0x4481982, .) /workspace/anago-v1.17.0- .go :281 +0x197 (0xc66b7bcfd0, 0xc005d24a3c, 0x2, 0x101, 0x1, 0x2, Gxc60b76aec8) /workspace/anago-v1.17.0-rc .2.10+70132b0F130acc/src/k8s .io/kubernetes/output/dockerized/go/src/k8s .io/kubernetes/vendor/k8s .io/apiextensions-apiserver/pkg/controller/openapi/builder/builder .go :146 +0x49d (0xc66b7bcfd0, 0x0, Oxb, Gx447f1db, 0x4, 0x1) /workspace/anago-v1.17.0- +0x1a8 ("Controller).Run(0xc0012c8050, Gxc0G1d61200, 0xc007a6d1e0, 0xc069cced80) /workspace/anago-v1.17.0- +0x2da created by /workspace/anago-v1.17.0- .go :236 +0x244 panic: runtime error: slice bounds out of range [-1:] [recovered] panic: runtime error: slice bounds out of range [-1:] Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign
There is a typo in the subject.


Question: flags.StringVar(&TestContext.ReportDir, "report-dir", "", "Path to the directory where the JUnit XML reports should be saved. Default is empty, which doesn't generate these reports.") flags.Var(cliflag.NewMapStringBool(&TestContext.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/experimental features.") flags.StringVar(&TestContext.ContainerRuntime, "container-runtime", "remote", "The container runtime of cluster VM instances (remote).") <del> flags.StringVar(&TestContext.ContainerRuntimeEndpoint, "container-runtime-endpoint", "unix:///var/run/dockershim.sock", "The container runtime endpoint of cluster VM instances.") </del> <ins> flags.StringVar(&TestContext.ContainerRuntimeEndpoint, "container-runtime-endpoint", "unix:///var/run/containerd/containerd.sock", "The container runtime endpoint of cluster VM instances.") </ins> flags.StringVar(&TestContext.ContainerRuntimeProcessName, "container-runtime-process-name", "dockerd", "The name of the container runtime process.") flags.StringVar(&TestContext.ContainerRuntimePidFile, "container-runtime-pid-file", "/var/run/docker.pid", "The pid file of the container runtime.") flags.StringVar(&TestContext.SystemdServices, "systemd-services", "docker", "The comma separated list of systemd services the framework will dump logs for.")

NPD e2e Tests wouldn't start 12/08 suggested it may be related to /cc No response No response /sig node /priority important-soon
do you think you can take a look?
checked the logs for the latest failure, failing with the below message while starting kubernetes systemd probe
/triage accepted /assign can you please follow up with on this?
Compared the changelog, there are no changes in the kubernetes/node-problem-detector on the failure date.
I suspect this issue might be from the k8s-staging image. let me know if there is any one that could take a look at this issue related to the image used in infra.
The following are my theories there must be some change in the ssh secret keys that is resulting in the unexpected commit digest Image has some changes related to manifest pulling in docker and containerd. k8s-staging image - issue.
/reopen still failing
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the test is green now. There was the recent PR from that fixed it


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/extensions/v1beta1:go_default_library", "//staging/src/k8s.io/api/policy/v1beta1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: registry = "k8s.gcr.io/build-image", repository = "debian-base", # Ensure the digests above are updated to match a new tag <del> tag = "buster-v1.2.0",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.3.0",  # ignored, but kept here for documentation </ins> ) container_pull(

What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


Question: const ( liveness probeType = iota readiness <ins> probeResultSuccessful string = "successful" probeResultFailed     string = "failed" probeResultUnknown    string = "unknown" </ins> ) // For debugging.

Currently the kubelet exports liveness/readiness probe metric information at an . The metric data looks like this: This is incredibly useful metric information, however, it would be better if these metrics were counters rather than gauges. Currently a success probe is registered as a '0' value and a failure is registered as a '1' value. However, depending on your prometheus scrape interval, you can lose bits of information. It would be better if we emitted these metrics as counters. /sig instrumentation node
Perhaps a new counters metrics to record the the number of failure is a better option. Because the gauge metrics is easy to watch and alert. I can do it if agree with it. /assign
This is a good point. However, if we really did want to use it for this purpose, it would be better if the gauge metric reported '0' for success or else the current running count of failures for a given prober endpoint. With the current implementation, we are needlessly dropping some bits. It may be useful to have both. One can provide a more real time signal for alerting but the other gives you better historical data on master component probe results.
do you have a preference on gauge vs counter here?
For our purposes a counter makes much more sense, because we want to use it to figure out the proportion of time something was healthy (e.g. for calculating availability SLOs). It also means we can have a much higher fidelity measurement - e.g. we could ramp up the healthchecks to every 5s but only scrape the results every minute. Can you point at some examples of people doing this, so I can better understand how these metrics are being used? It seems like using point-in-time samples of health is going to result in noisy alerts, unless you do more work elsewhere to aggregate ("alert if this metric was 0 for X minutes" - which then runs the risk of missing samples, as mentioned)
if the metrics value is 0 or 1, we can simple set prometheus rule as just like you mentioned.
Yes, I agree that the alert rule would be simple but it also feels prone to be noisy (as mentioned). A failed probe is not necessarily alert-worthy, since the kubelet defers action until the ; by default, the kubelet doesn't do anything until we have 3 consecutive failures.

Yep, so with in alert rule, it will not fire alert noisily. /cc
All the facts have already been stated, I'm in favor of removing the gauges and add counters (I don't see a state that we can't interpolate from the counter that the gauge displays today, yet the counter is more reliable in alerting aspects as mentioned).
sure. As a counter, how to represent successful probe? If the probe is successful, no action is taken, otherwise plus 1 to this metrics ?
I'm thinking something along the lines of:
Yeah, I was thinking along the same lines (i.e. using the result of the probe as a label).
Also now that we're talking about this and inspired by the metrics overhaul of 1.14, we should probably standardize these label values across Kubernetes. I'll add it to the next agenda.
A little off topic but wondering if someone could help guide me on how to access this endpoint? I've tried both k8s 1.13 and 1.14 and cannot access metrics/probes (i can only hit /metrics) via the api server http endpoint. Is it exposed somewhere else, or is it a flag I have to enable?
I had this issue for few days. What you should do is in your config you have to add "metricspath: /metrics/cadvisor". This will scrape out all container metrics that you would like to see. global: scrapeinterval: 5s evaluationinterval: 5s rulefiles: /etc/prometheus-rules/cpu- /etc/prometheus-rules/instance- scrapeconfigs: jobname: 'kubernetes-nodes' tlsconfig: cafile: bearertokenfile: kubernetessdconfigs: role: node metricspath: /metrics/cadvisor relabelconfigs: sourcelabels: [address] regex: '(.*):' replacement: '${1}:' targetlabel: address
This isn't an apiserver metric, it's actually a kubelet metric, so it's exposed on a different port. Are you using 8080? I think it should be or something like that.
@ munsayac13 thanks for the suggestion.

This is what I ended up doing to solve this: update the kubelet servicemonitor (e.g kubectl edit servicemonitor kubelet) this: bearerTokenFile: honorLabels: true interval: 30s path: /metrics/probes port: https-metrics scheme: https tlsConfig: insecureSkipVerify: true
Hi folks, I wonder if we could add the old gauge metrics back for backward compatibility sake. We just realized this change breaks some of our components that relies on the old metrics during the migration to v1.15.x.
We don’t have stability guarantees on these metrics (or any for that matter). I don’t think we’ll be reintroducing these, but we’re introducing a metrics stability framework for these things to not happen again or have the same deprecation process as kubernetes APIs in the future. Sorry for the inconvenience!
that is understandable.


Question: // See https://github.com/open-telemetry/opentelemetry-go/tree/main/example/passthrough return otelhttp.NewHandler(wrappedHandler, "KubernetesAPI", opts...) } <ins> func getSpanNameFromRequestInfo(info *request.RequestInfo, r *http.Request) string { spanName := "/" + info.APIPrefix if info.APIGroup != "" { spanName += "/" + info.APIGroup } spanName += "/" + info.APIVersion if info.Namespace != "" { spanName += "/namespaces/{:namespace}" } spanName += "/" + info.Resource if info.Name != "" { spanName += "/" + "{:name}" } if info.Subresource != "" { spanName += "/" + info.Subresource } return r.Method + " " + spanName } </ins>

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: }, nil } <ins> func (plugin *azureDataDiskPlugin) RequiresFSResize() bool { return true } func (plugin *azureDataDiskPlugin) ExpandVolumeDevice( spec *volume.Spec, newSize resource.Quantity, oldSize resource.Quantity) (resource.Quantity, error) { if spec.PersistentVolume == nil || spec.PersistentVolume.Spec.AzureDisk == nil { return oldSize, fmt.Errorf("invalid PV spec") } diskController, err := getDiskController(plugin.host) if err != nil { return oldSize, err } return diskController.ResizeDisk(spec.PersistentVolume.Spec.AzureDisk.DiskName, oldSize, newSize) } </ins> func (plugin *azureDataDiskPlugin) ConstructVolumeSpec(volumeName, mountPath string) (*volume.Spec, error) { mounter := plugin.host.GetMounter(plugin.GetPluginName()) pluginDir := plugin.host.GetPluginDir(plugin.GetPluginName())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: cc.Recorder, cc.ComponentConfig.AlgorithmSource, stopCh, <del> framework.NewRegistry(), </del> <ins> registry, </ins> cc.ComponentConfig.Plugins, cc.ComponentConfig.PluginConfig, scheduler.WithName(cc.ComponentConfig.SchedulerName),

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: FlagPersistent = 0x1 // FlagHashed specify IPVS service hash flag FlagHashed = 0x2 <ins> // FlagSourceHash enables IPVS service hashing on source port and source IP FlagSourceHash = 0x10 </ins> ) // Equal check the equality of virtual server.

The current support of the load distribution method and in kube-proxy ipvs mode only performs hashing of source IP address to select endpoints. As a result, all connections from a single client (i.e. from a particular source IP) are always forwarded to only one endpoint matching with the hash value of this source IP. If the client opens a vast majority of connections to the k8s cluster, it will result in traffic imbalance as only one endpoint located on one node handles the connections while the remaining endpoints/nodes are idle. Here is an example of the problem when using method for kube-proxy Deploy a service with 4 endpoints located on 4 nodes.Open 200 connections from a client to node IP of any node (e.g. telnet 192.168.1.1 ). All connections receive replies from only one single endpoint.The distribution printout also shows that all connections are forwarded to the endpoint (e.g. 1.0.4.2:5001 in this example)Similar issue is seen with method Connections from a client should be evenly distributed to all endpoints like to avoid unbalanced load distribution between cluster nodes and endpoints. This can be easily done by enabling the source port hash flag that IPVS supports for the methods which performs hashing on both source IP and source port when selecting the target endpoint.
/sig network /area kube-proxy /area ipvs
This issue was raised based on the discussion of the issue
triage/accepted /assign
/triage/accepted
/triage accepted (damn it!)


Question: <del> #!/bin/bash # Copyright 2014 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Generates pod and secret to deploy origin against configured Kubernetes provider set -o errexit set -o nounset set -o pipefail ORIGIN=$(dirname "${BASH_SOURCE}") KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/kubectl.sh" > /dev/null 2>&1 # Check all prerequisites are on the path HAVE_JQ=$(which jq) if [[ -z ${HAVE_JQ} ]]; then echo "Please install jq" exit 1 fi HAVE_BASE64=$(which base64) if [[ -z ${HAVE_BASE64} ]]; then echo "Please install base64" exit 1 fi # Capture information about your kubernetes cluster TEMPLATE="--template="{{ index . "current-context" }}"" CURRENT_CONTEXT=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) TEMPLATE="--template="{{range .contexts}}{{ if eq .name ${CURRENT_CONTEXT} }}{{ .context.cluster }}{{end}}{{end}}"" CURRENT_CLUSTER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) TEMPLATE="--template="{{range .contexts}}{{ if eq .name ${CURRENT_CONTEXT} }}{{ .context.user }}{{end}}{{end}}"" CURRENT_USER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) TEMPLATE="--template="{{range .clusters}}{{ if eq .name ${CURRENT_CLUSTER} }}{{ index . "cluster" "certificate-authority" }}{{end}}{{end}}"" CERTIFICATE_AUTHORITY=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) TEMPLATE="--template="{{range .clusters}}{{ if eq .name ${CURRENT_CLUSTER} }}{{ .cluster.server }}{{end}}{{end}}"" KUBE_MASTER=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) TEMPLATE="--template="{{range .users}}{{ if eq .name ${CURRENT_USER} }}{{ index . "user" "auth-path" }}{{end}}{{end}}"" AUTH_PATH=$( "${kubectl}" "${config[@]:+${config[@]}}" config view -o template "${TEMPLATE}" ) # Build an auth_path file to embed as a secret AUTH_PATH_DATA=$(cat ${AUTH_PATH} ) KUBE_USER=$( echo ${AUTH_PATH_DATA} | jq '.User' ) KUBE_PASSWORD=$( echo ${AUTH_PATH_DATA} | jq '.Password' ) KUBE_CERT_FILE=$( echo ${AUTH_PATH_DATA} | jq '.CertFile' ) KUBE_KEY_FILE=$( echo ${AUTH_PATH_DATA} | jq '.KeyFile' ) cat <<EOF >"${ORIGIN}/origin-auth-path" { "User": ${KUBE_USER}, "Password": ${KUBE_PASSWORD}, "CAFile": "/etc/secret-volume/kube-ca", "CertFile": "/etc/secret-volume/kube-cert", "KeyFile": "/etc/secret-volume/kube-key" } EOF # Collect all the secrets and encode as base64 ORIGIN_KUBECONFIG_DATA=$( cat ${ORIGIN}/origin-kubeconfig.yaml | base64 --wrap=0) ORIGIN_CERTIFICATE_AUTHORITY_DATA=$(cat ${CERTIFICATE_AUTHORITY} | base64 --wrap=0) ORIGIN_AUTH_PATH_DATA=$(cat ${ORIGIN}/origin-auth-path | base64 --wrap=0) ORIGIN_CERT_FILE=$( cat ${KUBE_CERT_FILE//"/} | base64 --wrap=0) ORIGIN_KEY_FILE=$( cat ${KUBE_KEY_FILE//"/}  | base64 --wrap=0) cat <<EOF >"${ORIGIN}/secret.json" { "apiVersion": "v1beta2", "kind": "Secret", "id": "kubernetes-secret", "data": { "kubeconfig": "${ORIGIN_KUBECONFIG_DATA}", "kube-ca": "${ORIGIN_CERTIFICATE_AUTHORITY_DATA}", "kube-auth-path": "${ORIGIN_AUTH_PATH_DATA}", "kube-cert": "${ORIGIN_CERT_FILE}", "kube-key": "${ORIGIN_KEY_FILE}" } } EOF echo "Generated Kubernetes Secret file: ${ORIGIN}/secret.json" # Generate an OpenShift Origin pod # TODO: In future, move this to a replication controller when we are not running etcd in container cat <<EOF >"${ORIGIN}/pod.json" { "apiVersion": "v1beta1", "id": "openshift", "kind": "Pod", "labels": {"name": "origin"}, "desiredState": { "manifest": { "containers": [ { "command": [ "start", "master", "--kubernetes=${KUBE_MASTER}", "--kubeconfig=/etc/secret-volume/kubeconfig", "--public-kubernetes=https://10.245.1.3:8443", "--public-master=https://10.245.1.3:8443", ], "image": "openshift/origin:latest", "imagePullPolicy": "PullIfNotPresent", "name": "origin", "ports": [ { "name": "https-api", "containerPort": 8443, "hostPort": 8443, }, { "name": "https-ui", "containerPort": 8444, "hostPort": 8444, } ], "volumeMounts": [ { "mountPath": "/etc/secret-volume", "name": "secret-volume", "readOnly": true } ] } ], "restartPolicy": { "never": {} }, "version": "v1beta2", "volumes": [ { "name": "secret-volume", "source": { "secret": { "target": { "kind": "Secret", "name": "kubernetes-secret", "namespace": "default" } } } } ] } } } EOF echo "Generated Kubernetes Pod file: ${ORIGIN}/pod.json" cat <<EOF >"${ORIGIN}/api-service.json" { "apiVersion": "v1beta1", "kind": "Service", "id": "origin-api", "port": 8443, "containerPort": 8443, "selector": { "name": "origin" }, } EOF echo "Generated Kubernetes Service file: ${ORIGIN}/api-service.json" cat <<EOF >"${ORIGIN}/ui-service.json" { "apiVersion": "v1beta1", "kind": "Service", "id": "origin-ui", "port": 8444, "containerPort": 8444, "selector": { "name": "origin" }, } EOF echo "Generated Kubernetes Service file: ${ORIGIN}/ui-service.json" </del>

This PR was prematurely merged: Need to update the PR based on the actual steps that were needed to get what was demonstrated at the public Kubernetes meet-up last week.
Any chance you can get rid of the hostPort from the example?
- depending on the Kubernetes provider, yes. For Vagrant based provider, I think I still need the hostPort because I do not have routes. For the GCE provider, I should be able to get rid of it because I have routes. In both providers, there were a number of setup steps that did not make the merged PR. For those that are tracking this issue (which I know folks have pinged me on separately) needs to be configured to allow privelaged containers needs to be configured to allow pushing to an insecure registry (this is point in time) Ideally, I could use a ReplicationController in the example to load balance OpenShift, but until we can provide a pattern to run etcd as a pod, I am unable to do this and need to use the embedded etcd we end up running.
Note: I will resolve this issue in 2 phases. Vagrant (which was demonstrated) to document using GCE (which has some additional steps on firewall rules)
Turns out OpenShift Origin is too secure now ;-) This is blocked until OpenShift Origin allows you to mint certs on the client, and then supply the certs on startup as a secret to the pod. This feature is coming soon. /cc
I now have all the required pieces to get this to work. I hope to get a PR submitted to fix this issue by end of this week.


Question: continue fi # Skip a directory if 1) it has no files and 2) all the subdirectories contain a go.mod file. <del> if [[ -z "$(find "${DEPS_DIR}/${PACKAGE}" -mindepth 1 -maxdepth 1 -type f)" ]]; then </del> <ins> if [[ -z "$(find "${DEPS_DIR}/${PACKAGE}/" -mindepth 1 -maxdepth 1 -type f)" ]]; then </ins> misses_go_mod=false while read -d "" -r SUBDIR; do if [[ ! -e "${SUBDIR}/go.mod" ]]; then misses_go_mod=true break fi <del> done < <(find "${DEPS_DIR}/${PACKAGE}" -mindepth 1 -maxdepth 1 -type d -print0) </del> <ins> done < <(find "${DEPS_DIR}/${PACKAGE}/" -mindepth 1 -maxdepth 1 -type d -print0) </ins> if [[ $misses_go_mod = false ]]; then echo "${PACKAGE} has no files, skipping" >&2 continue

What happened: Running manually includes this file in LICENSES. Running complains if it is present. The file was incorrectly dropped in This needs to be resolved before 1.19 ships /priority important-soon /milestone v1.19 /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
fixed in


Question: const ( // Agnhost image Agnhost = iota <del> // Alpine image Alpine </del> // APIServer image APIServer // AppArmorLoader image

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: jobs primarely, like: Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this commit merged: Testgrid link: Reason for failure: The test should always pull the image from (), it is not configurable. Anything else we need to know: /sig testing /sig windows /area conformance This issue was fixed once before:


Question: Start the server: ```sh <del> cluster/kubecfg.sh -proxy -www $PWD/www </del> <ins> cluster/kubectl.sh proxy -www=$PWD/www </ins> ``` The UI should now be running on [localhost](http://localhost:8001/static/index.html#/groups//selector)

It's causing problems and is kind-of GCE dependent anyway, we should just use the kubecfg binary, and document the environment variables it uses.
I think its beneficial to keep something, but remove the GCE dependent pieces, i.e. "external-ip" piece.
I put together some instructions here:
We can remove this when kubecfg is deprecated .
Reopened since it still needs to be removed. /cc
Doh. Didn't mean to declare this fixed in that PR.
Closing this (again) as it is a dupe of


Question: detect-project &> /dev/null export PATH=$(get_absolute_dirname $kubectl):$PATH kubectl="${GCLOUD}" <del> fi if [[ "$KUBERNETES_PROVIDER" == "vagrant" ]]; then # When we are using vagrant it has hard coded auth.  We repeat that here so that # we don't clobber auth that might be used for a publicly facing cluster. config=( "--auth-path=$HOME/.kubernetes_vagrant_auth" ) elif [[ "${KUBERNETES_PROVIDER}" == "gke" ]]; then </del> # GKE runs kubectl through gcloud. config=( "preview"

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: // File system resize was requested, proceed glog.V(4).Infof(volumeToMount.GenerateMsgDetailed("MountVolume.resizeFileSystem entering", fmt.Sprintf("DevicePath %q", volumeToMount.DevicePath))) <ins> if volumeToMount.VolumeSpec.ReadOnly { simpleMsg, detailedMsg := volumeToMount.GenerateMsg("MountVolume.resizeFileSystem failed", "requested read-only file system") glog.Warningf(detailedMsg) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FileSystemResizeFailed, simpleMsg) return nil } </ins> diskFormatter := &mount.SafeFormatAndMount{ Interface: mounter, Exec:      og.volumePluginMgr.Host.GetExec(expandableVolumePlugin.GetPluginName()),

We should not attempt resizing of file system if volume is mounted as read-only. /sig storage cc


Question: numErrs: 0, }, { <del> name: "invalid duplicate targetports (number with same protocol)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 1, }, { name: "invalid duplicate targetports (name with same protocol)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "r", Port: 2, Protocol: "TCP", TargetPort: intstr.FromString("http")}) }, numErrs: 1, }, { name: "valid duplicate targetports (number with different protocols)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromInt(8080)}) s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromInt(8080)}) }, numErrs: 0, }, { name: "valid duplicate targetports (name with different protocols)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeClusterIP s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "q", Port: 1, Protocol: "TCP", TargetPort: intstr.FromString("http")}) s.Spec.Ports = append(s.Spec.Ports, core.ServicePort{Name: "r", Port: 2, Protocol: "UDP", TargetPort: intstr.FromString("http")}) }, numErrs: 0, }, { </del> name: "valid type - cluster", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeClusterIP

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api /sig network What happened: Prior to 1.8 it was possible to specify the same targetPort in the service definition multiple times. In 1.8, validation was that prohibited this. What you expected to happen: I expect backwards compatibility for GA resources, so having the same targetPort multiple times should be allowed. This is a valid use case IMO for target ports. If you take the example below, the load balancer terminates SSL, so that manifest allows the nginx pods to receive http on port 80 while the load balancer serves traffic on 80 & 443. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Original ticket for validation:
cc
/kind bug /area api
Sounds reasonable. WDYT?
Hi, same problem here. Trying to create a service with helm, both 80 and 443 ports using same targetPort, and got this error: $ kubectl version
Feels reasonable, and it seems this is impacting enough people to warrant reverting the original change. WDYT?
Given Service objects are , how a breaking change like this was accepted?
/assign
For the one looking for a temporary way to fix it you have to mix named port with value, see below:
seems to be the PR that shipped this. Why would we want to validate that there are no duplicate target ports anyway? There should be plenty of IP addresses free inside the cluster IP ranges on most clusters (or you have a different problem on your hands) and there are plenty of high-range ports free for the NodePort type. The NAT path through the node should be safe. I'm not sure why this validation is needed. It seems perfectly safe and sane to me. Plus this is a pretty severe breaking change for the ingress-nginx sidecar for pods.


Question: } else { dv, err = ds.getDockerVersion() } <ins> if err != nil { return nil, err } </ins> apiVersion, err := semver.Parse(dv.APIVersion) if err != nil {

I was running an experiment to verify that Taints are correctly updated when Condition changes from Unreachable to NotReady and found out that Kubelet can't start when the docker is not running. I did an experiment on CVM and: made no op made no op - wait reverted changes to The result is following: Is this expected, or kubelet should normally start and report itself as 'NotReady'?
Thanks for reporting! kubelet shouldn't panic in this case. This is a bug, and I uploaded a PR to fix this.
We should cherry-pick the fix. On the side note, the impact of this bug at startup is limited because kubelet would fail to start if docker isn't running already.


Question: pod := entrypointTestPod() pod.Spec.Containers[0].Args = []string{"override", "arguments"} <del> testContainerOutput("override arguments", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override arguments", c, pod, []string{ </ins> "[/ep override arguments]", }) })

Testing how kubelet client rotation works, I discovered that on startup kubelet uses the cert in () and not the new cert in () This prevent kubelet to start kubelet uses the latest client certificate a 1 node cluster with kubespray (kubeadm should do) ntpd/chronyd the date 11 months in the future master certs kubelet that you have a new client cert the date 11 months more in the future and restart kubelet kubelet fails to start because it uses the cert embedded in instead of Both and are set, if I remove kubelet boots but fails to renew the cert Here an easy fix: Also I noticed that is RSA where is EC/P-256 Kubernetes version (use ): v1.18.15 (I can retest in more recent version but haven't seen anything in the release notes) Cloud provider or hardware configuration: VM OS (e.g: ): Kernel (e.g. ): Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle (not sure if this is more kubelet or kubeadm)
is the embedding certificates? see the second warning under here:
you can call on the same node to make the point to the rotatable file. newer versions of kubeadm do this change automatically.
i don't see a core bug here. i missed that you are using kubespray, which uses kubeadm. please report any issues with the kubespray issue tracker first: /kind support /close
Closing this issue. <detailsIn response to : i don't see a core bug here. please triage any issues with the kubespray issue tracker first: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks kubespray indeed doesn't call anywhere, I'll go look at how to fix that
like i said, newer versions of do this automatically.

this might be the result of an old cluster being upgrade using but the is remaining stale. the manual step in or calling the on a node is required.
It was a fresh deployment of k8s 1.18 / kubespray 2.15, I'm doing a fresh deployment of k8s 1.20 / kubespray master right now, but maybe kubespray is using kubeadm in weird ways / include old workarounds that leave us with having embedded certs instead of pointing to .
Found the culprit: Kubespray regenerate the conf files using kubeadm to make sure the server is up to date Thanks for the help and sorry for the spam
it would make sense to just include the kubelet-finalize phase in there too.
I prefer to just change the server using one of lineinfile/sed/kubectl so next time there is a new step like kubelet-finalize we don't break again.


Question: <ins> {% set kube_user = grains.kube_user -%} </ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"admin", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} <ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"{{kube_user}}", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} </ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubelet", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kube_proxy", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubecfg", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}}

abac-authz- hardcodes "admin" instead of using the $KUBE_USER variable. This means that "admin" is the only basic-auth username that will work with that abac config.
To workaround until it is fixed, you can add a line to the master's abac-authz- file: and then restart the kube-apiserver (docker kill it, and let it restart).


Question: kube::log::status "Deleting rs" kubectl delete rs frontend "${kube_flags[@]}" # Post-condition: no pods from frontend replica set <del> kube::test::get_object_assert 'pods -l "tier=frontend"' "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert 'pods -l "tier=frontend"' "{{range.items}}{{$id_field}}:{{end}}" '' </ins> ### Create and then delete a replica set with cascade=false, make sure it doesn't delete pods. # Pre-condition: no replica set exists kube::test::get_object_assert rs "{{range.items}}{{$id_field}}:{{end}}" '' # Command kubectl create -f hack/testdata/frontend-replicaset.yaml "${kube_flags[@]}" <ins> # wait for all 3 pods to be set up kube::test::wait_object_assert 'pods -l "tier=frontend"' "{{range.items}}{{$pod_container_name_field}}:{{end}}" 'php-redis:php-redis:php-redis:' </ins> kube::log::status "Deleting rs" kubectl delete rs frontend "${kube_flags[@]}" --cascade=false # Wait for the rs to be deleted.

/kind bug What happened: hang on "kubectl delete" command on my deployer machine, but can run other commands as usual ! can run "kubectl delete" command on master node, deployer and master have the same directory. What you expected to happen: delete some resource How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: Environment: Kubernetes version (use ): ! Cloud provider or hardware configuration: azure, I am not using cloud provider OS (e.g. from /etc/os-release): centos 7.5 Kernel (e.g. ): Linux cn2-k8s-deployer 3.10.0-862.3.3.el7.x8664 SMP Fri Jun 15 04:15:27 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: installed by kubespray Others:
/sig azure
this is caused by a switch to kubectl to use "foreground deletion" in 1.11.0 that relies on server-side garbage collection being responsive to clean up the resource and its dependents, and blocks until it does. the hang means that your controller manager isn't running, or the garbage collection controller isn't responsive at the moment you are deleting the object. because of interactions like this, kubectl is returning to use background deletion in 1.11.1 (see )
fixed in , will be in 1.11.1 /close
The fix in makes me sad :( We have just found out that foreground cascade deletion is not working in kubectl again. The workaround for that would be not to wait, kubectl has a flag for that: Personally I would prefer to get a hanging kubectl in that case. was keeping a backward compatibile behavior the main reason to switch back to ? We should probably have a flag for explicit at least then... Otherwise there is no way to trigger a foreground deletion via kubectl :(
the hang means that your controller manager isn't running, or the garbage collection controller isn't responsive at the moment you are deleting the object. numerous bug reports suggest you are in the minority :) that behavior also means that with just an apiserver running, always hangs, which doesn't seem proper.

to be clear, kubectl still waits until the object you requested to be deleted is actually deleted, which was the priority for scripting compatibility (to make delete-then-recreate cases work properly) the way client-side reapers did cascading deletion in kubectl was foreground out of necessity, since client-side deletion is not re-entrant otherwise. it was only implemented that way for resources which had client-side reapers, which was a small hard-coded set. blocking client comments on controller behavior unnecessarily is problematic (as numerous reports of issues with the switch to foreground deletion revealed). I would not object to an option to run deletion in foreground mode, but it doesn't make for a good default.
the deletion mode is implemented in GC on the server side, and while understand the use case for client-side reapers, I disagree that it is only useful for some client-side blocking. The real use case we have (as I previously described in ) is building a tree of objects where deleting a root will force deletion of all its children first and will block the deletion of itself in the meantime. This has nothing to do with a client-side logic, it's just annoying that won't ever use this mode, and our users obviously deleting objects via . This is very similar to namespace behavior: deleted namespace gets blocked until all objects within it are gone. I understand the concerns of having foreground deletion mode enabled by default though.


Question: return fmt.Errorf("failed to delete disk(%s) since it's in attaching or detaching state", diskURI) } <del> rerr := c.common.cloud.DisksClient.Delete(ctx, resourceGroup, diskName) </del> <ins> disk, rerr := c.common.cloud.DisksClient.Get(ctx, resourceGroup, diskName) if rerr != nil { return rerr.Error() } if disk.ManagedBy != nil { return fmt.Errorf("disk(%s) already attached to node(%s), could not be deleted", diskURI, *disk.ManagedBy) } rerr = c.common.cloud.DisksClient.Delete(ctx, resourceGroup, diskName) </ins> if rerr != nil { return rerr.Error() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: check disk status before disk azure disk There is little possibility that attach disk would cost 10min, and disk could be attached, deleted in same time, that would cause race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: } if cfg.Disk.SCSIControllerType == "" { <del> cfg.Disk.SCSIControllerType = DefaultSCSIControllerType </del> <ins> cfg.Disk.SCSIControllerType = LSILogicSASControllerType </ins> } else if !checkControllerSupported(cfg.Disk.SCSIControllerType) { glog.Errorf("%v is not a supported SCSI Controller type. Please configure 'lsilogic-sas' OR 'pvscsi'", cfg.Disk.SCSIControllerType) return nil, errors.New("Controller type not supported. Please configure 'lsilogic-sas' OR 'pvscsi'")

vSphere cloud provider volume plugin attach disk limits the number of volumes because of two issues: new SCSI controller is only created when one does not exist. A new controller should be when required and limit not reached. disk will fail when attaching disk to slot 7 on controller. Slot 7 on SCSI controller is reserved and therefore should be skipped. This issue has been brought up in Govmomi library .


Question: return } <ins> type PingCommand string const ( IPv4PingCommand PingCommand = "ping" IPv6PingCommand PingCommand = "ping6" ) </ins> // CheckConnectivityToHost launches a pod to test connectivity to the specified // host. An error will be returned if the host is not reachable from the pod. // // An empty nodeName will use the schedule to choose where the pod is executed. <del> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, timeout int) error { </del> <ins> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, pingCmd PingCommand, timeout int) error { </ins> contName := fmt.Sprintf("%s-container", podName) command := []string{ <del> "ping", </del> <ins> string(pingCmd), </ins> "-c", "3", // send 3 pings "-W", "2", // wait at most 2 seconds for a reply "-w", strconv.Itoa(timeout),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: e2e tests provide only an (IPv4) ping test for external connectivity. What you expected to happen: Need a way to conditionally run a ping6 external connectivity check, and disable the (IPv4) external connectivity check, for IPv6-only clusters. How to reproduce it (as minimally and precisely as possible): Examine the e2e code ((), or run this test on an IPv6-only cluster. Anything else we need to know?: This test will be needed for IPv6 gating CI tests, at least until Kubernetes dual-stack is fully implemented. /area ipv6 Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network


Question: continue } if strings.Contains(line, spec) { <del> parts := strings.Split(filter, " ") </del> // expected tc line: <del> // filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 if len(parts) != 19 { return classAndHandleList, false, fmt.Errorf("unexpected output from tc: %s %d (%v)", filter, len(parts), parts) </del> <ins> // `filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1` (old version) or // `filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 not_in_hw` (new version) matches := classAndHandleMatcher.FindStringSubmatch(filter) if len(matches) != 3 { return classAndHandleList, false, fmt.Errorf("unexpected output from tc: %s %d (%v)", filter, len(matches), matches) </ins> } <del> resultTmp := []string{parts[18], parts[9]} </del> <ins> resultTmp := []string{matches[2], matches[1]} </ins> classAndHandleList = append(classAndHandleList, resultTmp) } }

What happened: I tried to set up bandwidth limiting by booting a cluster with , which by default sets up kubenet. I used the annotations xM xM and then used to test the ingress and egress bandwidths, with no effect. Does anyone know if this feature is still supported in kubenet? And if so what are the steps I need to take to boot a cluster that can use pod annotations to do tc shaping. What you expected to happen: Per traffic shaping should be implemented on Kubenet as of 4/12/2018. Anything else we need to know?: There are a lot of issues regarding traffic shaping not working. None have a concrete answer of how to get it to work on kubenet: Relevant: I believe this is where it was first implemented. Environment: Kubernetes version (use ): 1.11.8 Cloud provider or hardware configuration: AWS CC:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/triage unresolved I am a bot run by
can we put this on sig-network agenda? the discussion item would be: what is the current status of this feature (alpha, beta, GA?) what should we do next with this API?
I have this to the agenda, will grab some info.
AFAIK traffic shaping is still supported in kubenet. Does a pod with these annotations come up successfully but has no traffic shaping applied? Or do things fail earlier? If it comes up successfully, could you run ?
On GKE we do not officially support this, our default image even don't have tc installed. I tried with ubuntu, the kubenet is failed to parse the tc output. kubenet expect. // filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 tc return.// filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 notinhw
If this issue has been triaged, please comment .

Meta: /athenabot mark-triage-reminder I am a bot run by
Depending on which CNI plugin you use you need to install the bandwidth plugin, see:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
since it is in progress /priority critical-urgent /milestone v1.17
/lifecycle frozen
/assign
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
The fix is still pending in review.
Will it be merged by this Thursday for v1.17 code freeze?
/milestone v1.18 (v1.17 code freeze)
matching assigned priority on PR
Hello I'm one of the Bug Triage shadows for the 1.18 release. Since this issue is tagged for 1.18, I'd like to check its status. Is this issue intended for milestone 1.18? Friendly reminder: Code freeze is scheduled for March 5th.
Hello Checking in again. Do you think this will make it in for 1.18 release?
Hey Checking in again as this issue is marked as . Are you close to completing this for v1.18 release? Another reminder that code freeze is just around the corner! (March 5th).
Friendly reminder that code freeze is in 2 days and this issue is marked as
The fix got approved by ,should be able to catch the freeze.


Question: elif [[ ${platform} == "linux/ppc64le" ]]; then export CGO_ENABLED=1 export CC=powerpc64le-linux-gnu-gcc <ins> elif [[ ${platform} == "darwin/"* ]]; then # See https://github.com/kubernetes/kubernetes/issues/32999 export GOROOT=${K8S_PATCHED_GOROOT} </ins> fi fi }

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: elif [[ ${platform} == "linux/ppc64le" ]]; then export CGO_ENABLED=1 export CC=powerpc64le-linux-gnu-gcc <ins> elif [[ ${platform} == "darwin/"* ]]; then # See https://github.com/kubernetes/kubernetes/issues/32999 export GOROOT=${K8S_PATCHED_GOROOT} </ins> fi fi }

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: KubeConfig:              flag.NewStringFlag("/var/lib/kubelet/kubeconfig"), ContainerRuntimeOptions: *NewContainerRuntimeOptions(), CertDirectory:           "/var/run/kubernetes", <del> CloudProvider:           v1alpha1.AutoDetectCloudProvider, </del> RootDirectory:           v1alpha1.DefaultRootDir, <ins> // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations // with cloud providers instead of in the core repo. // More details here: https://github.com/kubernetes/kubernetes/issues/50986 CloudProvider: v1alpha1.AutoDetectCloudProvider, </ins> } }

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: ErrPodNotMatchHostName = NewPredicateFailureError("HostName", "node(s) didn't match the requested hostname") // ErrPodNotFitsHostPorts is used for PodFitsHostPorts predicate error. ErrPodNotFitsHostPorts = NewPredicateFailureError("PodFitsHostPorts", "node(s) didn't have free ports for the requested pod ports") <del> // ErrServiceAffinityViolated is used for CheckServiceAffinity predicate error. ErrServiceAffinityViolated = NewPredicateFailureError("CheckServiceAffinity", "node(s) didn't match service affinity") </del> // ErrNodeUnderMemoryPressure is used for NodeUnderMemoryPressure predicate error. ErrNodeUnderMemoryPressure = NewPredicateFailureError("NodeUnderMemoryPressure", "node(s) had memory pressure") // ErrNodeUnderDiskPressure is used for NodeUnderDiskPressure predicate error.

Part of See example /sig scheduling /kind cleanup /help /priority important-soon
/assign /remove-help


Question: // UpdateApplyAnnotation calls CreateApplyAnnotation if the last applied // configuration annotation is already present. Otherwise, it does nothing. func UpdateApplyAnnotation(info *resource.Info, codec runtime.Encoder) error { <del> if original, err := GetOriginalConfiguration(info); err != nil || len(original) <= 0 { </del> <ins> if original, err := GetOriginalConfiguration(info.Mapping, info.Object); err != nil || len(original) <= 0 { </ins> return err } return CreateApplyAnnotation(info, codec)

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: // At this point there was either no PDB or we succeeded in decrementing // Try the delete <del> _, _, err = r.store.Delete(ctx, eviction.Name, eviction.DeleteOptions) </del> <ins> deletionOptions := eviction.DeleteOptions if deletionOptions == nil { // default to non-nil to trigger graceful deletion deletionOptions = &metav1.DeleteOptions{} } _, _, err = r.store.Delete(ctx, eviction.Name, deletionOptions) </ins> if err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set.

Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


Question: if err != nil || len(internalDNS) == 0 { //TODO: It would be nice to be able to determine the reason for the failure, // but the AWS client masks all failures with the same error description. <del> glog.V(2).Info("Could not determine private DNS from AWS metadata.") </del> <ins> glog.V(4).Info("Could not determine private DNS from AWS metadata.") </ins> } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNS}) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig aws What happened: Every few seconds I get spam in my logs about "Could not determine public DNS from AWS metadata" from every node. This is really very spammy. What you expected to happen: No spam. How to reproduce it (as minimally and precisely as possible): Launch a private topology cluster with kops. Anything else we need to know?: There will never be a public-hostname in the metadata for private topology clusters. This line: Should be glog.V(4) i suspect, like line 1033 above it. Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Jessie Kernel (e.g. ): 4.4.78-k8s Install tools: kops Others:
fixed in . PTAL. Thanks!
Thanks On a side note the metadata should probably be cached to stop thrashing the API but I don't know enough to be sure.
/cc


Question: func (r RequiredIPVSKernelModulesAvailableCheck) Check() (warnings, errors []error) { glog.V(1).Infoln("validating the kernel module IPVS required exists in machine or not") <ins> kernelVersion, ipvsModules, err := GetKernelVersionAndIPVSMods(r.Executor) if err != nil { errors = append(errors, err) } </ins> // Find out loaded kernel modules out, err := r.Executor.Command("cut", "-f1", "-d", " ", "/proc/modules").CombinedOutput() if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: } if len(tried) != 0 { <del> return fmt.Errorf("recognized csr %q as %v but subject access review was not approved", csr.Name, tried) </del> <ins> return certificates.IgnorableError("recognized csr %q as %v but subject access review was not approved", csr.Name, tried) </ins> } return nil

Each node waiting for bootstrapping creates 1 line per 2-3s in the logs when the requesting client doesn't have access. It should not be logging at v(0), these are all v(4) level. We should be reporting these appropriately. I don't want a 5k node cluster showing this.
I'd like to fix it


Question: // rbdInfo runs `rbd info` command to get the current image size in MB. func (util *RBDUtil) rbdInfo(b *rbdMounter) (int, error) { var err error <del> var output string var cmd []byte </del> <ins> var output []byte </ins> // If we don't have admin id/secret (e.g. attaching), fallback to user id/secret. id := b.adminId

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kubectl create -f ceph- and result is ok now i want to resize pv from 1G to 2G, edit kubectl describe pvc pvc-expand-test What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: ceph storageclass yaml: ceph secret: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:41:50Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration:OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708 (Core) Kernel (e.g. ): Linux m01 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: ceph version 13.2.2 () mimic (stable) <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What's the version of your ceph cluster? k8s uses the command to get rbd image's size, and it expects the size unit in the output is . However, in your environment the unit is :
ceph version 13.2.2 () mimic (stable)
It seems that the output format is different between different ceph versions.

I will try to fix it.
/assign
your PR has been merged to master branch, and when will Kubernetes v1.11 fix this bug?
Sorry for the late, I've send cherry-pick PRs:
It seems this change is not working in some cases: I think this is due to two concurrent conditions: 1- ceph client still complain on stderr about not being able to load config file even if there is no need as a key is supplied and the result is (hence the output on stderr of ). This happens not only in hyperkube. 2- it seems we capture both stdout and stderr from the execution of the command (please note the output in the error from ) and then there is then a problem trying to unmarshall the JSON (to be clear, the unuseful warning is on stderr, the json output on stdout). I am able to have a workaround creating empty files for and so that the ceph client does not output anymore but I feel it would be much better to parse only stdout. A reference on the source code for Thanks edit:typo
could you tell where did you put the empty and ceph.keyring files? On the node running kubelet?
Inside the pod of the as the rbd resize is executed by this component.
I had the same problem, I ceph. conf and ceph. Keyring and solved it, but I think k8s got the default user of CEPH before going back to ceph. conf and ceph. Keyring. The following code validates my conjecture: //num 672 Ultimately, I want k8s to get those two files through configurable CEPH parameters. What do you think?
as far as I know you do not need to configure those two files if you use a key. That is still an hypothesis as in the version of Ceph we are using this switch seems to be undocumented: Anyway the warning can be safely ignored and I do not see why we should parse when there is no error
I two files with empty content before, no error; later I a soft connection called and ceph. Keyring (pointing to the ceph configuration file and key ring generated by my cluster), but ceph did not report an error. . I think ceph only needs two files to exist and doesn't care about its contents.

Fortunately, the current measures can solve the problem.
this matches my experience (see above, I created the empty files too and it gives no error) but another possibility is just not parse which I find "cleaner" as there is no error in this case even from the error code, and even what we get is just a warning.
Same issue as here. The warnings about the missing etc files break the JSON output. Adding empty files in the container fixes resizing.
Shouldn't this be reopened until the bug is fixed? Creating the two files also "solves" the issue for us.
/reopen cc
Reopened this issue. <detailsIn response to : /reopen cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
which version of ceph cluster is in use in your setup ?
We are running a ceph cluster with . kube-controller-manager is running which internally uses . Also just to prevent any confusion: I've used to resize the PVC, not rancher.
This also seems related to .
We are running
Are there plans to resolve this issue?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Answering myself - apparently it has been fixed in k8s 1.20:


Question: } func TestDescribePod(t *testing.T) { <del> deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(10, 0, 0)} </del> <ins> deletionTimestamp := metav1.Time{Time: time.Now().UTC().AddDate(-10, 0, 0)} </ins> gracePeriod := int64(1234) condition1 := corev1.PodConditionType("condition1") condition2 := corev1.PodConditionType("condition2")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubectl describe pv xxx shows last<invalidif pv is terminating ! I found pvc and pod have the same issue What you expected to happen: Terminating (lasts 10s) How to reproduce it (as minimally and precisely as possible): 1 create a pv, pvc or pod is terminating 2 using kubectk describe xxx Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: }, gvk: examplev1.SchemeGroupVersion.WithKind("Pod"), expectedObj: &examplev1.Pod{ <ins> TypeMeta: metav1.TypeMeta{ APIVersion: "example.apiserver.k8s.io/v1", Kind:       "Pod", }, </ins> ObjectMeta: metav1.ObjectMeta{ Name: "pod1", Labels: map[string]string{

The admission webhook review requests seem not to include a GVK in the raw object. It's included elsewhere in the review request, so it's generally not an issue, except that the unstructured codec requires GVKs to be present (it's not a technical requirement AFAICT, just a check done in the codec logic). This means that trying to use or on an Unstructured fails. A workaround in the mean time is to use , but it would be nice if this just worked properly (fixes to the review request logic and the unstructured codec logic. <details</summary</details<details</summary</detailsserving CA and certs (at and ) webhook with CA bundle `kubectl create -f that the output object doesn't have GVK in the field Environment: Kubernetes version (use ): master-ish () /sig api-machinery
when encoded to the wire, apiVersion and kind must always be included. do you have a gist handy of what the webhook receives for a built-in type (like a pod) and a for a custom resource?
Looks like custom objects (held in unstructured objects) would include GVK, but in-tree types would not. It might be as simple as explicitly setting the GVK in ConvertToGVK: and we need to update the webhook test to ensure objects in requests always have an apiVersion/kind
fixed in
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving original configuration from:n%vnfor:", info), info.Source, err) } <ins> // Create the versioned struct from the original from the server for // strategic patch. // TODO: Move all structs in apply to use raw data. Can be done once // builder has a RawResult method which delivers raw data instead of // internal objects. versionedObject, _, err := decoder.Decode(current, nil, nil) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("converting encoded server-side object back to versioned struct:n%vnfor:", info), info.Source, err) } </ins> // Compute a three way strategic merge patch to send to server. <del> patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, info.VersionedObject, true) </del> <ins> patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) </ins> if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfrom:n%vnfor:" return cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current, info), info.Source, err)

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: type cacheWatcher struct { sync.Mutex copier  runtime.ObjectCopier <del> input   chan *watchCacheEvent </del> <ins> input   chan watchCacheEvent </ins> result  chan watch.Event done    chan struct{} filter  watchFilterFunc

Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.


Question: return fmt.Errorf("Failed to resize replication controller %s to zero: %v", name, err) } <del> if err := wait.Poll(time.Second, time.Minute*20, client.ControllerHasDesiredReplicas(c, rc)); err != nil { return fmt.Errorf("Error waiting for replication controller %s replicas to reach 0: %v", name, err) </del> <ins> // Wait up to 20 minutes until all replicas are killed. endTime := time.Now().Add(time.Minute * 20) for { if time.Now().After(endTime) { return fmt.Errorf("Timeout while waiting for replication controller %s replicas to 0", name) } remainingTime := endTime.Sub(time.Now()) err := wait.Poll(time.Second, remainingTime, client.ControllerHasDesiredReplicas(c, rc)) if err != nil { glog.Errorf("Error while waiting for replication controller %s replicas to read 0: %v", name, err) } else { break } </ins> } // Delete the replication controller.

In Jenkins scalability suite, Density test is failing on cleanup: Expected error: <*errors.errorString | 0xc20d27c460: { s: "Error waiting for replication controller my-hostname-density3000--e362-11e4-90b1- replicas to reach 0: Get http://127.0.0.1:4001/v2/keys/registry/controllers/default/my-hostname-density3000--e362-11e4-90b1-?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: cannot assign requested address", } Error waiting for replication controller my-hostname-density3000--e362-11e4-90b1- replicas to reach 0: Get http://127.0.0.1:4001/v2/keys/registry/controllers/default/my-hostname-density3000--e362-11e4-90b1-?quorum=false&recursive=false&sorted=false: dial tcp 127.0.0.1:4001: cannot assign requested address not to have occurred We need some retrying while waiting for rc to be deleted (similar to what I've done for waiting for all replicas in )
This just happened again on Jenkins.


Question: }, }, }, <ins> { name: "set new condition which doesn't have lastTransitionTime set", crdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, }, newCondition: CustomResourceDefinitionCondition{ Type:    Established, Status:  ConditionFalse, Reason:  "NotAccepted", Message: "Not accepted", }, expectedcrdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionFalse, Reason:             "NotAccepted", Message:            "Not accepted", LastTransitionTime: metav1.Date(2018, 1, 2, 0, 0, 0, 0, time.UTC), }, }, }, { name: "append new condition which doesn't have lastTransitionTime set", crdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, }, newCondition: CustomResourceDefinitionCondition{ Type:    Terminating, Status:  ConditionFalse, Reason:  "NeverEstablished", Message: "resource was never established", }, expectedcrdCondition: []CustomResourceDefinitionCondition{ { Type:               Established, Status:             ConditionTrue, Reason:             "Accepted", Message:            "the initial names have been accepted", LastTransitionTime: metav1.Date(2018, 1, 1, 0, 0, 0, 0, time.UTC), }, { Type:               Terminating, Status:             ConditionFalse, Reason:             "NeverEstablished", Message:            "resource was never established", LastTransitionTime: metav1.Date(2018, 2, 1, 0, 0, 0, 0, time.UTC), }, }, }, </ins> } for _, tc := range tests { crd := generateCRDwithCondition(tc.crdCondition)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign