Question: // KubeletExtraArgs passes through extra arguments to the kubelet. The arguments here are passed to the kubelet command line via the environment file // kubeadm writes at runtime for the kubelet to source. This overrides the generic base-level configuration in the kubelet-config-1.X ConfigMap // Flags have higher priority when parsing. These values are local and specific to the node kubeadm is executing on. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> KubeletExtraArgs map[string]string `json:"kubeletExtraArgs,omitempty"` // IgnorePreflightErrors provides a slice of pre-flight errors to be ignored when the current node is registered.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: Name:              serviceName, Namespace:         namespace, CreationTimestamp: metav1.NewTime(time.Now()), <del> Annotations:       map[string]string{managedBySetupAnnotation: managedBySetup}, </del> }, Spec: v1.ServiceSpec{ Ports:    []v1.ServicePort{{TargetPort: intstr.FromInt(80)}},

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: # By default, linux wraps base64 output every 76 cols, so we use 'tr -d' to remove whitespaces. # Note 'base64 -w0' doesn't work on Mac OS X, which has different flags. <del> CA_KEY_BASE64=$(base64 "${CERT_DIR}/pki/private/ca.key" | tr -d 'rn') CA_CERT_BASE64=$(base64 "${CERT_DIR}/pki/ca.crt" | tr -d 'rn') MASTER_CERT_BASE64=$(base64 "${CERT_DIR}/pki/issued/${MASTER_NAME}.crt" | tr -d 'rn') MASTER_KEY_BASE64=$(base64 "${CERT_DIR}/pki/private/${MASTER_NAME}.key" | tr -d 'rn') KUBELET_CERT_BASE64=$(base64 "${CERT_DIR}/pki/issued/kubelet.crt" | tr -d 'rn') KUBELET_KEY_BASE64=$(base64 "${CERT_DIR}/pki/private/kubelet.key" | tr -d 'rn') KUBECFG_CERT_BASE64=$(base64 "${CERT_DIR}/pki/issued/kubecfg.crt" | tr -d 'rn') KUBECFG_KEY_BASE64=$(base64 "${CERT_DIR}/pki/private/kubecfg.key" | tr -d 'rn') KUBEAPISERVER_CERT_BASE64=$(base64 "${CERT_DIR}/pki/issued/kube-apiserver.crt" | tr -d 'rn') KUBEAPISERVER_KEY_BASE64=$(base64 "${CERT_DIR}/pki/private/kube-apiserver.key" | tr -d 'rn') </del> <ins> CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/private/ca.key" | tr -d 'rn') CA_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/ca.crt" | tr -d 'rn') MASTER_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/issued/${MASTER_NAME}.crt" | tr -d 'rn') MASTER_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/private/${MASTER_NAME}.key" | tr -d 'rn') KUBELET_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/issued/kubelet.crt" | tr -d 'rn') KUBELET_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/private/kubelet.key" | tr -d 'rn') KUBECFG_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/issued/kubecfg.crt" | tr -d 'rn') KUBECFG_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/private/kubecfg.key" | tr -d 'rn') KUBEAPISERVER_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/issued/kube-apiserver.crt" | tr -d 'rn') KUBEAPISERVER_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CERT_DIR}/pki/private/kube-apiserver.key" | tr -d 'rn') </ins> # Setting up an addition directory (beyond pki) as it is the simplest way to # ensure we get a different CA pair to sign the proxy-client certs and which # we can send CA public key to the user-apiserver to validate communication. <del> AGGREGATOR_CA_KEY_BASE64=$(base64 "${AGGREGATOR_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') REQUESTHEADER_CA_CERT_BASE64=$(base64 "${AGGREGATOR_CERT_DIR}/pki/ca.crt" | tr -d 'rn') PROXY_CLIENT_CERT_BASE64=$(base64 "${AGGREGATOR_CERT_DIR}/pki/issued/proxy-client.crt" | tr -d 'rn') PROXY_CLIENT_KEY_BASE64=$(base64 "${AGGREGATOR_CERT_DIR}/pki/private/proxy-client.key" | tr -d 'rn') </del> <ins> AGGREGATOR_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${AGGREGATOR_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') REQUESTHEADER_CA_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${AGGREGATOR_CERT_DIR}/pki/ca.crt" | tr -d 'rn') PROXY_CLIENT_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${AGGREGATOR_CERT_DIR}/pki/issued/proxy-client.crt" | tr -d 'rn') PROXY_CLIENT_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${AGGREGATOR_CERT_DIR}/pki/private/proxy-client.key" | tr -d 'rn') </ins> # Setting up the Kubernetes API Server Konnectivity Server auth. # This includes certs for both API Server to Konnectivity Server and # Konnectivity Agent to Konnectivity Server. <del> KONNECTIVITY_SERVER_CA_KEY_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') KONNECTIVITY_SERVER_CA_CERT_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/ca.crt" | tr -d 'rn') KONNECTIVITY_SERVER_CERT_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') KONNECTIVITY_SERVER_KEY_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/server.key" | tr -d 'rn') KONNECTIVITY_SERVER_CLIENT_CERT_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/issued/client.crt" | tr -d 'rn') KONNECTIVITY_SERVER_CLIENT_KEY_BASE64=$(base64 "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/client.key" | tr -d 'rn') KONNECTIVITY_AGENT_CA_KEY_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') KONNECTIVITY_AGENT_CA_CERT_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/ca.crt" | tr -d 'rn') KONNECTIVITY_AGENT_CERT_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') KONNECTIVITY_AGENT_KEY_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/server.key" | tr -d 'rn') KONNECTIVITY_AGENT_CLIENT_CERT_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/issued/client.crt" | tr -d 'rn') KONNECTIVITY_AGENT_CLIENT_KEY_BASE64=$(base64 "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/client.key" | tr -d 'rn') CLOUD_PVL_ADMISSION_CA_KEY_BASE64=$(base64 "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') CLOUD_PVL_ADMISSION_CA_CERT_BASE64=$(base64 "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/ca.crt" | tr -d 'rn') CLOUD_PVL_ADMISSION_CERT_BASE64=$(base64 "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') CLOUD_PVL_ADMISSION_KEY_BASE64=$(base64 "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/private/server.key" | tr -d 'rn') </del> <ins> KONNECTIVITY_SERVER_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') KONNECTIVITY_SERVER_CA_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/ca.crt" | tr -d 'rn') KONNECTIVITY_SERVER_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') KONNECTIVITY_SERVER_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/server.key" | tr -d 'rn') KONNECTIVITY_SERVER_CLIENT_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/issued/client.crt" | tr -d 'rn') KONNECTIVITY_SERVER_CLIENT_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_SERVER_CERT_DIR}/pki/private/client.key" | tr -d 'rn') KONNECTIVITY_AGENT_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') KONNECTIVITY_AGENT_CA_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/ca.crt" | tr -d 'rn') KONNECTIVITY_AGENT_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') KONNECTIVITY_AGENT_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/server.key" | tr -d 'rn') KONNECTIVITY_AGENT_CLIENT_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/issued/client.crt" | tr -d 'rn') KONNECTIVITY_AGENT_CLIENT_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${KONNECTIVITY_AGENT_CERT_DIR}/pki/private/client.key" | tr -d 'rn') CLOUD_PVL_ADMISSION_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/private/ca.key" | tr -d 'rn') CLOUD_PVL_ADMISSION_CA_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/ca.crt" | tr -d 'rn') CLOUD_PVL_ADMISSION_CERT_BASE64=$(base64 $BASE64_INPUT_FLAG "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/issued/server.crt" | tr -d 'rn') CLOUD_PVL_ADMISSION_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "${CLOUD_PVL_ADMISSION_CERT_DIR}/pki/private/server.key" | tr -d 'rn') </ins> } # Set up easy-rsa directory structure.

under kubernetes project and running get error: ! I am running on darwin arm64 running get this: if running on linux:
/area test
/sig cli
/sig testing /area test-infra /remove-sig cli /remove-area test
FWIW You pretty much have to install gnu coreutils to develop here, many scripts will actually refuse to run if the gnu version of utils are not available. That said, we should fix this.
/triage accepted


Question: "hostPath":         initHostpath, "hostPathSymlink":  initHostpathSymlink, "emptyDir":         initEmptydir, <del> "gcePD":            initGCEPD, </del> <ins> "gcePDPVC":         initGCEPD, </ins> "gcePDPartitioned": initGCEPDPartition, "nfs":              initNFS, "nfsPVC":           initNFSPVC,

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "net/http" ) // WithCacheControl sets the Cache-Control header to "no-cache, private" because all servers are protected by authn/authz. // see https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching#defining_optimal_cache-control_policy func WithCacheControl(handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { // Set the cache-control header if it is not already set if _, ok := w.Header()["Cache-Control"]; !ok { w.Header().Set("Cache-Control", "no-cache, private") } handler.ServeHTTP(w, req) }) } </ins>

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: - -k - --cache-size=1000 - --no-negcache <ins> - --dns-loop-detect </ins> - --log-facility=- - --server=/$DNS_DOMAIN/127.0.0.1#10053 - --server=/in-addr.arpa/127.0.0.1#10053

Is this a BUG REPORT or FEATURE REQUEST?: What happened: If hosts points to a local DNS server, or if it contains the cluster's DNS server (to enable looking up services from host), it'll cause DNS loop which can effectively disable kube-dns or make it very slow. What you expected to happen: At least, kube-dns should detect localhost or cluster DNS server entries in and not send queries to them. How to reproduce it (as minimally and precisely as possible): Use a system with local cashing DNS server, or add cluster's DNS server address to hosts Anything else we need to know?: According to documentation, adding should make dnsmasq detect that and disable the upstream DNS server which cause the loop. It should prevent DNS from becoming unresponsive in the first case (a better solution is to extract real upstream servers, which probably cannot be done for every configuration); and for the second case it should skip sending queries to itself and send to real upstream servers.
/sig network
Could you describe the use-case more? If the host's contains cluster DNS server and some other upstream server - like 8.8.8.8, will services lookups work reliably from the host? If the services lookup goes to the 8.8.8.8 server, it will return NXDOMAIN. If it hits the clusterDNS server, only then it would work, correct?
Also ref If the first server returns NXDOMAIN, the second server will not be tried. How do you make both cluster DNS name and non-cluster DNS name resolvable on the host?
If cluster DNS server comes before a real DNS server, it works fine. Also note that what man page says is the glibc (or libc) behavior, not other users (e.g. dnsmasq sends its queries to any one of them, but can be configured to query them in strict order ( flag), or to query all of them (). Anyway, consider when cluster DNS server comes before the real server: cluster DNS queries, the cluster DNS server responds with the IP of the cluster or pod. external names, the query is sent to cluster DNS server. It doesn't find the name, so it tries to send it to upstream DNS servers; which in my example are cluster DNS server itself and an external DNS server.

If dnsmasq sends it to external DNS server, it'll receive the answer and can serve the query. If it sends it to itself, there is a loop and the same process happens again until dnsmasq decides to query the other server (which might not happen!). WIth flag, when dnsmasq sends the query to itself, it'll recognize the loop and will remove its own address from the list of upstream DNS servers, so it'll always send external names to the external DNS server from now on.
Thanks for the explanation. I believe it might work with explicit configuration. Though adding a flag to upstream for fulfilling a not quite common use case seems a bit odd to me. Have you thought of configuring upstream DNS servers via kube-dns config [1] such that it will not send request to itself at all? [1] -
Note that the flag has two use cases, the other one is prevent kube-dns breakage when upstream DNS server is a local one (e.g. If you deploy to Ubuntu, which has local caching DNS server by default (e.g. in 18.04, it is systemd-resolved and contains 127.0.0.53)). Actually I guess there are a number of DNS related issues which might be solved by this flag too. So, while this flag will not make kubernetes magically find real upstream dns servers in such cases, it prevents kube-dns crash; which has caused frustration and some time for me to discover. So, either kubelet should error out and exit if the given contains a local address, or such flag should be used by kube-dns to prevent DNS loops. And, IIRC, the only reason that cluster's DNS server is not to host, at least by kubespray, is mentioned to be host specific in the docs. So, my use case seems to be an ideal one and it would be the default one if it was easy to setup for different platforms. About using kube-dns config, no. We're going to provide automatic kubernetes deployment for many environments, and so we don't know the upstream DNS servers upfront, and we don't want to hard code it. We want it to be the DNS servers used in the target network. So, configuring a specific upstream DNS server is a no-go.


Question: return &node.Status, nil } <ins> func (p *PodCache) clearNodeStatus() { p.lock.Lock() defer p.lock.Unlock() p.currentNodes = map[objKey]api.NodeStatus{} } </ins> // TODO: once Host gets moved to spec, this can take a podSpec + metadata instead of an // entire pod? func (p *PodCache) updatePodStatus(pod *api.Pod) error {

I have 3 identical nodes, 1 replication controller with 2 replicas of a pod. I simulate failure of one of the nodes to where pods have been scheduled by simply unplugging the node. Kubernetes realizes that the node has failed, but the pod gets not re-scheduled to the another available node. The documentation at says the following: and So is that a mistake in the documentation or a missing or a bug? I use v0.10.
That's definitely a bug, and something that used to work correctly. I will investigate today, and follow up on this bug. Thanks for the report. Brendan On Feb 5, 2015 8:11 AM, "peter-braun" wrote:
Actually, it's really a missing feature -- . I believe the last of the mechanisms we needed were ~last week. said he would start on it.
Is there a rudimentary solution that we could get in quickly, and then improve next week?
Just sent a simple PR with the fix.
Note that this issue is about a replication-controller set of pods. This isn't which is a broader "fix pods on failed nodes in general" issue.
Ah, a cache bug. Will take a look.
Re-opening this, because I reverted the relevant code in the replication controller with
I'll comment on the other thread later. Sorry I was out yesterday. I'm a bit free this weekend and should be able to work on the problem for at least one day.
Fixed by . Re-open if not fixed.
was merged a few days ago. It referenced , but not this issue.


Question: } func addRunFlags(cmd *cobra.Command) { <del> cmd.Flags().String("generator", "", "The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'.") </del> <ins> cmd.Flags().String("generator", "", "The name of the API generator to use.  Default is 'deployment/v1beta1' if --restart=Always, otherwise the default is 'job/v1'.  This will happen only for cluster version at least 1.2, for olders we will fallback to 'run/v1' for --restart=Alwyas, 'run-pod/v1' for others.") </ins> cmd.Flags().String("image", "", "The image for the container to run.") cmd.MarkFlagRequired("image") cmd.Flags().IntP("replicas", "r", 1, "Number of replicas to create for this container. Default is 1.")

Tested with: Most of the simple basic starting guides direct users to create a cluster with . If I run kubectl 1.2 against a 1.1 cluster, I'll get the following behavior now I happen to know I can get the old behavior by doing but I don't expect an average user to know that or be able to find it easily. I'm mainly concerned that we will be breaking recommended flow for GKE users if they happen to use a new kubectl before their cluster's master is upgraded. We can just update our docs, but breaking a "porcelain" command seems like poor form. Other options include defaulting to the generator for a few patch releases, and/or making do some kind of version check so it can fallback to the generator if deployments/jobs are not available. thoughts?
We should update those docs and document how to achieve stable behavior: .
This would ideally use API discovery, though, to check whether the API exists.
How hard would it be to check whether Deployment existed? We had the same issue in 1.1, btw, with Job.
I'd accept a hack, also, which checked the k8s version if there wasn't enough discovery info in 1.1.
We have the same issue with Job in 1.2, also.
Fix is in


Question: switch pattern.VolType { case testpatterns.PreprovisionedPV: if pattern.VolMode == v1.PersistentVolumeBlock && !isBlockSupported { <del> It("should fail to create pod by failing to mount volume", func() { </del> <ins> It("should fail to create pod by failing to mount volume [Slow]", func() { </ins> init() defer cleanup()

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: import ( "net" "strconv" <ins> "strings" "fmt" "k8s.io/apimachinery/pkg/util/version" "k8s.io/utils/exec" </ins> ) // Interface is an injectable interface for running ipvs commands.  Implementations must be goroutine-safe.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: } func (eic *execInContainer) SetDir(dir string) { <del> //unimplemented </del> <ins> // unimplemented </ins> } func (eic *execInContainer) SetStdin(in io.Reader) { <del> //unimplemented </del> <ins> // unimplemented </ins> } func (eic *execInContainer) SetStdout(out io.Writer) {

I was trying a scenario where liveness/readiness probe command run inside the container for long time('sleep 10 && exit 10') with shorter timeoutSecod value say 2 Sec. As expected container restarted because of timeout but when i check the events in pod describe it stated only unhealthy and in events actual reason for probe failure is not printed "liveness probe failed:<'no reason printed'". Without clear logs or pod events description it is very difficult to debug and understand what is the failure. Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " ! It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " Below is the example manifest file to reproduce the issue.

apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 Create a pod using above manifest file and do pod describe (kubectl describe pod <pod-name ) and check for the events. ! No response kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:59:53Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:52:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} openstack <details</details<details</detailscontainerd No response
I tried to fix one similar issue in
/sig node
/triage accepted
can I work on this issue? Thanks very much. /assign
We've experienced this issue with clusters on v1.20. The for that version mention the following change: Since v1.20 is seemingly when exec probe timeouts started being enforced, would it be possible to have this fix for this issue backported to all minor versions since then? Thanks!
bump on that question
What do you think of? Thanks very much.


Question: CreateWithEventNamespace(event *v1.Event) (*v1.Event, error) // UpdateWithEventNamespace is the same as a Update, except that it sends the request to the event.Namespace. UpdateWithEventNamespace(event *v1.Event) (*v1.Event, error) <ins> // PatchWithEventNamespace is the same as a Patch, except that it sends the request to the event.Namespace. </ins> PatchWithEventNamespace(event *v1.Event, data []byte) (*v1.Event, error) // Search finds events about the specified object Search(scheme *runtime.Scheme, objOrRef runtime.Object) (*v1.EventList, error)

Follow up from We should cover events expansion - both for core v1: events v1beta: with unit tests - can you please take care of it?
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: try, I, want, to, have, a. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I want to have a try Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { m := &MyReq{req} switch { <ins> case m.isFor("GET", "/nodes"): return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: objBody(codec, oldNode)}, nil </ins> case m.isFor("GET", "/nodes/node-name"): return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: objBody(codec, oldNode)}, nil case m.isFor("PATCH", "/nodes/node-name"):

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: err = m.removeContainer(containerID) assert.NoError(t, err) <del> // Verify container log is removed assert.Equal(t, []string{expectedContainerLogPath, expectedContainerLogPathRotated, expectedContainerLogSymlink}, fakeOS.Removes) </del> <ins> // Verify container log is removed. // We could not predict the order of `fakeOS.Removes`, so we use `assert.ElementsMatch` here. assert.ElementsMatch(t, []string{expectedContainerLogSymlink, expectedContainerLogPath, expectedContainerLogPathRotated}, fakeOS.Removes) </ins> // Verify container is removed assert.Contains(t, fakeRuntime.Called, "RemoveContainer") containers, err := fakeRuntime.ListContainers(&runtimeapi.ContainerFilter{Id: containerID})

modified container log file removal in a way that made TestRemoveContainer flaky See Note that the flake is masked by the fact that the unit test job will retry tests up to 3 times. With , this flake is quite pronounced: A fix for the flake is in , but it's unclear to me whether it is correct that the removal order is non-deterministic. /assign /sig node /milestone v1.19 /priority critical-urgent
cc


Question: nodeObj, err := nsu.nodeLister.Get(string(nodeName)) if errors.IsNotFound(err) { // If node does not exist, its status cannot be updated. <del> // Remove the node entry from the collection of attach updates, preventing the // status updater from unnecessarily updating the node. </del> <ins> // Do nothing so that there is no retry until node is created. </ins> glog.V(2).Infof( "Could not update node status. Failed to find node %q in NodeInformer cache. Error: '%v'", nodeName, err) <del> nsu.actualStateOfWorld.RemoveNodeFromAttachUpdates(nodeName) </del> continue } else if err != nil { // For all other errors, log error and reset flag statusUpdateNeeded

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: Deployment running with read-only GCE PD. A node containing one of the deployment pods is deleted and recreated soon after. New pod is scheduled to the same node, but is stuck in ContainerCreating state. The node's kubelet logs show continuous "Volume is not yet attached according to node" errors. What you expected to happen: Pod recreated successfully. Anything else we need to know?: Cause by PR The issue happens as follows: When the node is deleted, GCE PD is detached from the cloud provider. AttachDetachController discovers the PD has been detached, and marks it as detached. The original pod still exists in the API server (i.e. before garbage collection), so AttachDetachController reattaches the PD, successfully. Before the node recovers, ADC tries to update the node status but finds that the node doesn't exist. Following the logic introduced in it never attempts to update the node status again. The node recovers, a new pod is scheduled to it, waits for a successful attach signal in the node status but never gets it. /cc /sig storage
/assign


Question: framework.Skipf("unable to run test without dynamic kubelet config enabled.") } // because adminssion Priority enable, If the priority class is not found, the Pod is rejected. <del> systemCriticalPriority := &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: systemCriticalPriorityName}, Value: scheduling.SystemCriticalPriority} </del> <ins> node := getNodeName(f) </ins> // Define test pods nonCriticalGuaranteed := getTestPod(false, guaranteedPodName, v1.ResourceRequirements{ Requests: v1.ResourceList{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: SchemaProps: spec.SchemaProps{ Description: "RawExtension is used to hold extensions in external versions.nnTo use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.nn// Internal package: type MyAPIObject struct {ntruntime.TypeMeta `json:",inline"`ntMyPlugin runtime.Object `json:"myPlugin"`n} type PluginA struct {ntAOption string `json:"aOption"`n}nn// External package: type MyAPIObject struct {ntruntime.TypeMeta `json:",inline"`ntMyPlugin runtime.RawExtension `json:"myPlugin"`n} type PluginA struct {ntAOption string `json:"aOption"`n}nn// On the wire, the JSON will look something like this: {nt"kind":"MyAPIObject",nt"apiVersion":"v1",nt"myPlugin": {ntt"kind":"PluginA",ntt"aOption":"foo",nt},n}nnSo what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)", Type:        []string{"object"}, <del> Properties: map[string]spec.Schema{ "Raw": { SchemaProps: spec.SchemaProps{ Description: "Raw is the underlying serialization of this object.", Type:        []string{"string"}, Format:      "byte", }, }, }, Required: []string{"Raw"}, </del> }, }, }

The Service Catalog includes the type in . The openapi schema generated for it is wrong. It has two problems: it includes the field, which is marked as required it doesn't allow additional properties to be included under it Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If the Service Catalog openapi schema is exposed to kubectl (this currently requires a few tweaks and a patch), it reports validation errors when creating a ServiceInstance resource. What you expected to happen: The schema should allow to include arbitrary properties and not require the property to be included. How to reproduce it (as minimally and precisely as possible): not easy at the moment (I'll add instructions later if I'm unable to fix this myself) Anything else we need to know?: I'd like to tackle this problem myself, but I need guidance. Should openapi-gen handle this automatically for all occurences of RawExtension, or should each field of type be json-tagged individually? Environment: Kubernetes version (use ):
I've heard you're the goto guy for the openapi-gen. Please help? :)
/sig api-machinery
OK, I've found I can specify a custom schema for RawExtension the way it's done in Time and MicroTime. The remaining problem is how to specify that the RawExtension object can be of any type. Using is not supported currently - if I use it in the schema, the API server dies with:
/assign /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
We've just hit this. /remove-lifecycle rotten /lifecycle frozen
hitting this
According to this: the raw extension can define its openapi type as: or alternatively what do you think?

do you still work on this or you want me to send a PR?
Would changing the definition of RawExtension to this help the situation? (adding ) I noticed that this removes Raw from the openapi defintion of it, without changing how it is marshalled or unmarshalled (since we have custom functions for that) I'll try making a PR for this


Question: return err } <ins> func newPodOnNode(podName, nodeName string, image string) *api.Pod { return &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: podName, Labels: map[string]string{ "name": podName, }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  podName, Image: image, Ports: []api.ContainerPort{{ContainerPort: 9376}}, }, }, NodeName:      nodeName, RestartPolicy: api.RestartPolicyNever, }, } } func createServeHostnamePodOnNode(c *client.Client, namespace, podName, nodeName string) error { pod, err := c.Pods(namespace).Create(newPodOnNode(podName, nodeName, serveHostnameImage)) if err == nil { Logf("Created pod %s on node %s", pod.ObjectMeta.Name, nodeName) } else { Logf("Failed to create pod %s on node %s: %s", podName, nodeName, err) } return err } </ins> func newReplicationControllerWithNameSelector(name string, replicas int, image string) *api.ReplicationController { return &api.ReplicationController{ ObjectMeta: api.ObjectMeta{

Master tracking bug:
The scenario of the test is the following: a service an RC with as many pods as the number of nodes for pods created, running and responding the first node and get the pod from it ... It is assumed that the first node has a pod - that's because kubernetes is supposed to distribute pods for which there is a service evenly among nodes. So we expect that each node has one pod. This turns out to be false. (in 80% cases the first node doesn't have a pod.
Thanks . cc: Is it a valid assumption that pods should be evenly spread across nodes? I assume not? If even spreading is a valid assumption, then there's a bug in the system. If not, then we need to fix this test.
To clarify the above, it's perfectly sensible to me that k8s should in an ideal world spread pods that constitute a service across as many nodes as possible. But I doubt whether our scheduler has such logic yet?
Our spreading is only a probabilistic guarantee. So this is incorrect test behavior.
We do actually have spreading logic. To determine if this is a scheduler bug, you'd have to know the state of the system more exactly. Spreading is balanced with resource usage, for example-- if something big was a little slow leaving the node, it could cause it to not get a pod. Or if it was briefly unavailable.
OK, I'd suggest that we change the test then, to not assume that pods are perfectly evenly spread across nodes.
, is there a way to run the scheduler with debug logs? I'm interested about the scores for nodes. (even if it won't help me in this issue, it's a nice exercise) I suppose the test fails because CPU utilization has a significant impact on scheduling pods - more that having a service. That's why all pods are scheduled on the same node. I'm going to remove the requirement that there is one pod per node (I'll still create a service though - just to make pods spread more evenly). I'll simply remove from the network one node that has a pod (without assuming that the first one has). I believe the scenario will still test everything we need ( please correct me if I'm wrong). The PR should be ready today.

By the way, a side effect of this test is that in a 2-node cluster, all add-ons will end up on one node, occupying 0.9 cores - this setup will stay for further tests, so it's good to be aware of it. I'll write this comment in the test code as well.
is ready for review. Possible second round will be needed. No changes in production code have been made (tests only).
I think if you set v=10 you'll see the log messages with that info in them.
This scenario is OK.
, sorry for such simple questions, but how do I set v=10?
you need to edit and change to in the command line args. The kubelet should then restart the container (validate with ) if it doesn't, should fix it.
Status: waiting for review of


Question: "context" "fmt" "io" <ins> "math/rand" </ins> "reflect" "sort" "strings" "testing" "github.com/aws/aws-sdk-go/aws" <ins> "github.com/aws/aws-sdk-go/aws/awserr" </ins> "github.com/aws/aws-sdk-go/service/ec2" "github.com/aws/aws-sdk-go/service/elb" <ins> "github.com/aws/aws-sdk-go/service/elbv2" </ins> "github.com/stretchr/testify/assert" "github.com/stretchr/testify/mock" "github.com/stretchr/testify/require" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/intstr" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using a pod like the nginx-ing-cntlr with an AWS Network Load Balancer (NLB), and then draining the node its running on to force a migration to another node pool, does not reliably deregister the drained nodes from the NLB Target group. This causes the nodeport used by the NLB to remain open, and the drained node ends up continuing to funnel traffic for other pods cause of being used. This leads to connection timeouts for clients, and downtime on migrations. Switching to a Classic ELB, instead of an NLB, with connection draining enabled does work, and the instances that are drained are practically removed immediately from the instances targeted by the ELB, as expected. There seems to be some sort of issue with NLB's as this is not happening, or happening rarely (only saw it once in many tests but couldn't recreate it). The node + LB set seems to be working as I see in the controller-mgr logs that the nodes are updated as I went from 11 nodes to 8, after attempting to drain + migrate a nodepool of 3 nodes with nginx-ing-cntlr all running on them: However, this update does not seem to be translating into deregistering instances from the NLB. And i've waited minutes & hours in case timeouts were playing a hand, but that didn't seem to help. What you expected to happen: On node draining, with being used, that the NLB would deregister the instances as expected. How to reproduce it (as minimally and precisely as possible): Launch a Deployment + NLB service on a node and drain it to force it over to another node pool. Watch the instances in the AWS NLB Target group never get out of "healthy" status or be drained + removed. Anything else we need to know?: Environment: AWS EKS Kubernetes version (use ): 1.12 & 1.13 Cloud provider or hardware configuration: AWS OS (e.g: ): AWS Linux 2 Kernel (e.g.

): 4.14.123-111.109.amzn2.x8664 Install tools: Network plugin and version (if this is a network-related bug): core-dns v1.2.6 Others:
/sig aws
Related:
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Possible resources include pods (po), replication controllers (rc), services (svc), nodes, events (ev), component statuses (cs), limit ranges (limits), <del> minions (mi), persistent volumes (pv), persistent volume claims (pvc) </del> <ins> nodes (no), persistent volumes (pv), persistent volume claims (pvc) </ins> or resource quotas (quota). By specifying the output as 'template' and providing a Go template as the value

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.


Question: // add non-defaults Force:   &force, Timeout: &timeout, <ins> Wait:    &wait, </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: } // Perform a bind mount to the full path to allow duplicate mounts of the same disk. options := []string{"bind"} <del> if disk.readOnly { </del> <ins> if disk.ReadOnly { </ins> options = append(options, "ro") } err = mounter.Mount(globalPDPath, volPath, "", options)

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: imageutils.GetE2EImage(imageutils.Perl), imageutils.GetE2EImage(imageutils.Nonewprivs), imageutils.GetPauseImageName(), <del> getGPUDevicePluginImage(), </del> imageutils.GetE2EImage(imageutils.NodePerfNpbEp), imageutils.GetE2EImage(imageutils.NodePerfNpbIs), imageutils.GetE2EImage(imageutils.NodePerfTfWideDeep),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig node /triage accepted


Question: corev1.ResourceMemory, memoryReqs.String(), int64(fractionMemoryReqs), memoryLimits.String(), int64(fractionMemoryLimits)) w.Write(LEVEL_1, "%st%s (%d%%)t%s (%d%%)n", corev1.ResourceEphemeralStorage, ephemeralstorageReqs.String(), int64(fractionEphemeralStorageReqs), ephemeralstorageLimits.String(), int64(fractionEphemeralStorageLimits)) extResources := make([]string, 0, len(allocatable)) <ins> hugePageResources := make([]string, 0, len(allocatable)) </ins> for resource := range allocatable { <del> if !resourcehelper.IsStandardContainerResourceName(string(resource)) && resource != corev1.ResourcePods { </del> <ins> if resourcehelper.IsHugePageResourceName(resource) { hugePageResources = append(hugePageResources, string(resource)) } else if !resourcehelper.IsStandardContainerResourceName(string(resource)) && resource != corev1.ResourcePods { </ins> extResources = append(extResources, string(resource)) } } sort.Strings(extResources) <ins> sort.Strings(hugePageResources) for _, resource := range hugePageResources { hugePageSizeRequests, hugePageSizeLimits, hugePageSizeAllocable := reqs[corev1.ResourceName(resource)], limits[corev1.ResourceName(resource)], allocatable[corev1.ResourceName(resource)] fractionHugePageSizeRequests := float64(0) fractionHugePageSizeLimits := float64(0) if hugePageSizeAllocable.Value() != 0 { fractionHugePageSizeRequests = float64(hugePageSizeRequests.Value()) / float64(hugePageSizeAllocable.Value()) * 100 fractionHugePageSizeLimits = float64(hugePageSizeLimits.Value()) / float64(hugePageSizeAllocable.Value()) * 100 } w.Write(LEVEL_1, "%st%s (%d%%)t%s (%d%%)n", resource, hugePageSizeRequests.String(), int64(fractionHugePageSizeRequests), hugePageSizeLimits.String(), int64(fractionHugePageSizeLimits)) } </ins> for _, ext := range extResources { extRequests, extLimits := reqs[corev1.ResourceName(ext)], limits[corev1.ResourceName(ext)] w.Write(LEVEL_1, "%st%st%sn", ext, extRequests.String(), extLimits.String())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: scheduler doesn't assign pod because 'not enough resource: hugepages-1Gi',but the node info show it has enough resource. describe node What you expected to happen: the pod was in normal status like pending and running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It was change default hugepage size to 1Gi and hugepages=36 numastat was show there has enough hugepage also proc/meminfo was show there has enough resource other info Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
cc do you have thoughts here? I see huge pages and think of you :)
Hmm. Are you sure that there are no other pods using huge page memory ? To me, it looks like the node has (huge page @ 1GiB) memory, you are requsting , but is already used. Can you verify that by listing the pods on the node, and check if some of them are using huge page memory already?
Also, the OS will report that no huge pages are in use as long as no processes have started using the hugepages. Eg. when a pod requesting with is assigned to a node, k8s will then mark the s with huge page memory it as "used". The cgroup controller (if hugetlb cgroup is mounted) will then set the usage limit to . Before the pod actually starts using the huge page memory, the OS will report that no huge page memory is used. When a process inside the pod maps with huge page memory via eg. , the OS will see that the memory is in used. If the process tries to map more than the requested amount, the map will fail (given that the hugetlb cgroup is mounted). To verify if the hugetlb cgroup is mounted, run .

If mounted, it will retrun something like this:
I’m so sorry that there actually has a other pod using hugepagei only see node info available hugepage. so i close the issuethanks a lot
No problem I think this issue is a symptom that something should be done to make it easier to understand. I will look into it.


Question: OperationCreateVolumeWithRawVSANPolicy = "CreateVolumeWithRawVSANPolicyOperation" ) <ins> var vCenterMetric *vcenterMetric func init() { vCenterMetric = &vcenterMetric{ vCenterInfos: make(map[string]types.AboutInfo), mux:          sync.Mutex{}, } } </ins> // vsphereAPIMetric is for recording latency of Single API Call. var vsphereAPIMetric = metrics.NewHistogramVec( &metrics.HistogramOpts{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: Name      string ID        string Labels    map[string]string <ins> // ExternalIPs should be false for Conformance test // Don't check nodeport on external addrs in conformance test, but in e2e test. ExternalIPs bool </ins> } // NewTestJig allocates and inits a new TestJig.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: name = "cri-tools-data-%s" % goarch, package_dir = "/usr/bin", tags = ["manual"], <del> deps = ["@cri_tools_%s//file" % goarch], </del> <ins> deps = ["@cri_tools_linux-%s//file" % goarch], </ins> ) for goarch in NODE_PLATFORMS["linux"]] [k8s_deb(

Right now cri-tools only include tarballs for linux /cc /cc
/sig release
/area release-eng /milestone v1.16 /help /priority important-soon
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /area release-eng /milestone v1.16 /help /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello! I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 2 days from now). Do we still target this issue to be fixed for 1.16?
No, nobody is working on this for 1.16. cc
/milestone clear
anything we can do in release team to move this forward?
I'm working on this one now if nobody else minds. :) /assign


Question: github.com/robfig/cron => github.com/robfig/cron v1.1.0 github.com/rogpeppe/fastuuid => github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af github.com/rogpeppe/go-internal => github.com/rogpeppe/go-internal v1.3.0 <del> github.com/rubiojr/go-vhd => github.com/rubiojr/go-vhd v0.0.0-20160810183302-0bfd3b39853c </del> <ins> github.com/rubiojr/go-vhd => github.com/rubiojr/go-vhd v0.0.0-20200706105327-02e210299021 </ins> github.com/russross/blackfriday => github.com/russross/blackfriday v1.5.2 github.com/russross/blackfriday/v2 => github.com/russross/blackfriday/v2 v2.0.1 github.com/satori/go.uuid => github.com/satori/go.uuid v1.2.0

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: func TestSafeMakeDir(t *testing.T) { defaultPerm := os.FileMode(0750) + os.ModeDir <ins> maxPerm := os.FileMode(0777) + os.ModeDir </ins> tests := []struct { name string // Function that prepares directory structure for the test under given

What happened: Adding a VolumeMount to a container with a multi-part subpath results in only the leaf directory having 777 permissions. If you define e.g. then within the volume, and will have permissions, whereas will have permissions. This means that if you want to remove and eventually from within a pod, you will need to use a privileged pod. If the sub-directories already exist, their permissions are unchanged, and the order in which these subpaths are created appears to be the order in which they appear within the spec -- if we see first, then will be created with 755 permissions and will be created with 777 permissions, even if there's another volumeMount for further down. What you expected to happen: Sub-directories created to support a volumeMount should all have the same permissions as the leaf directory. How to reproduce it (as minimally and precisely as possible): the yaml below. The first example has volume mounts so that is created before ; the second example has volume mounts in reversed order, so is encountered before , which results in being unwritable. <details<summaryThis pod will complete</summary</details<details<summaryThis pod will fail</summary</detailsAnything else we need to know?: I believe this is the return of Environment: Reproduces on minikube and a (OpenShift) development cluster I have access to Kubernetes version (use ): minikube: OpenShift cluster: output: Cloud provider or hardware configuration: Minikube: v1.15.1; docker driver; hostpath volumes (all default) OpenShift cluster: AWS, EBS volumes OS (e.g: ): Minikube: Fedora 32 OpenShift cluster: RHEL 8.2 Kernel (e.g. ): Minikube: OpenShift cluster: Install tools: minikube v1.15.1 / Not sure Network plugin and version (if this is a network-related bug): N/A Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig storage
/assign
GitHub didn't allow me to assign the following users: mengjiao-liu. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: }, }, options: printers.GenerateOptions{}, <del> // Columns: Name, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job4", "0/1", "20m", "10y"}}}, </del> <ins> // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job4", "Running", "0/1", "20m", "10y"}}}, }, { job: batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:              "job5", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, Spec: batch.JobSpec{ Completions: nil, }, Status: batch.JobStatus{ Succeeded: 0, Conditions: []batch.JobCondition{ { Type:   batch.JobComplete, Status: api.ConditionTrue, }, }, }, }, options: printers.GenerateOptions{}, // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job5", "Complete", "0/1", "", "0s"}}}, }, { job: batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:              "job6", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, Spec: batch.JobSpec{ Completions: nil, }, Status: batch.JobStatus{ Succeeded: 0, Conditions: []batch.JobCondition{ { Type:   batch.JobFailed, Status: api.ConditionTrue, }, }, }, }, options: printers.GenerateOptions{}, // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job6", "Failed", "0/1", "", "0s"}}}, }, { job: batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:              "job7", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, Spec: batch.JobSpec{ Completions: nil, }, Status: batch.JobStatus{ Succeeded: 0, Conditions: []batch.JobCondition{ { Type:   batch.JobSuspended, Status: api.ConditionTrue, }, }, }, }, options: printers.GenerateOptions{}, // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job7", "Suspended", "0/1", "", "0s"}}}, }, { job: batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:              "job8", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, Spec: batch.JobSpec{ Completions: nil, }, Status: batch.JobStatus{ Succeeded: 0, Conditions: []batch.JobCondition{ { Type:   batch.JobFailureTarget, Status: api.ConditionTrue, }, }, }, }, options: printers.GenerateOptions{}, // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job8", "FailureTarget", "0/1", "", "0s"}}}, }, { job: batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:              "job9", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, DeletionTimestamp: &metav1.Time{Time: time.Now().Add(1.9e9)}, }, Spec: batch.JobSpec{ Completions: nil, }, }, options: printers.GenerateOptions{}, // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job9", "Terminating", "0/1", "", "0s"}}}, </ins> }, }

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: // controllerName is a unique value used with LabelManagedBy to indicated // the component managing an EndpointSlice. controllerName = "endpointslice-controller.k8s.io" <del> // managedBySetupAnnotation is set on a Service to indicate that // EndpointSlices for the Service have already been configured with // LabelManagedBy. If this annotation is not set, all related EndpointSlices // will have LabelManagedBy set to reference this controller if the label // is not already set. Once all EndpointSlices are labeled, the Controller // will set this annotation on the Service. managedBySetupAnnotation = "endpointslice.kubernetes.io/managed-by-setup" // managedBySetupCompleteValue represents the value of the // managedBySetupAnnotation that indicates that the setup process has been // completed for a Service. managedBySetupCompleteValue = "true" </del> ) // NewController creates and initializes a new Controller

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: if err != nil { return err } <del> if o.Timeout == 0 { o.Timeout = kubectl.Timeout + time.Duration(10*len(pods))*time.Second } </del> err = o.deleteOrEvictPods(pods) if err != nil { pendingPods, newErr := o.getPodsForDeletion()

In writing some documentation for I noticed some issues with the implementation. I think the simplest user experience is that the command doesn't return until all the user pods are drained off the machine. In particular this means wait until all graceful termination is finished infinitely retry evictions that are rejected due to insufficient PodDisruptionBudget The current implementation does not do this, in particular wait for graceful termination appears to be at 10 sec number of pods, which may be much shorter than the max graceful termination we only retry evictions number of times My suggestion is to change the behavior so that we compute how long to wait for graceful termination, by taking max(grace periods of all pods that we want to delete) (or the value of --grace-period command line argument instead, if that is specified), and wait exactly that amount of time plus a small buffer we retry evictions infinitely WDYT? I think we should do this in 1.5 because the current behavior is very non-obvious.
The problem here is how long should we wait, if the user doesn't specify a timeout. The default timeout used in drain is following how long the reapers have in , , and . Maybe this is not long enough. I can make the changes. Do we still need exponential back off?
To summarize discussion with there are two possible approaches (pseudocode below) 1) Goroutine approach: For each pod P that meets the , do the following: 2) Non-goroutine approach: where "not timed out yet" means you haven't finished waiting max(--grace period argument to kubectl drain, max graceful termination spec across all pods on the node)
Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
The PR for this issue is waiting to be merged.
marking this as a release blocker due to being marked as such. Please update if needed cc: