Question: "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/images" "k8s.io/kubernetes/test/e2e/framework" <ins> e2enode "k8s.io/kubernetes/test/e2e/framework/node" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature"

Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.

Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage


Question: "//test/integration:all-srcs", "//test/kubemark:all-srcs", "//test/list:all-srcs", <del> "//test/soak/cauldron:all-srcs", </del> "//test/soak/serve_hostnames:all-srcs", "//test/typecheck:all-srcs", "//test/utils:all-srcs",

We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) Apparently the repo is referenced for docker push and pull in It's unclear to me whether we have broken anything, or if this is dead code. I would use to find out if this is referenced anywhere (aside from sundry vendor dirs) If dead code: remove it If used: move to test/e2e/images and follow the same pattern there used to push to /sig testing /area test /wg k8s-infra /priority important-soon /help /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : We've deleted all repos from Docker Hub and will be retiring the org/user (ref: ) /area test /wg k8s-infra /priority important-soon /good-first-issue I believe this is a pretty straightforward issue and am happy to review whatever PR's are made for it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil, nil } <ins> func serviceOwnsPublicIP(pip *network.PublicIPAddress, clusterName, serviceName string) bool { if pip != nil && pip.Tags != nil { serviceTag := pip.Tags[serviceTagKey] clusterTag := pip.Tags[clusterNameKey] if serviceTag != nil && *serviceTag == serviceName { // Backward compatible for clusters upgraded from old releases. // In such case, only "service" tag is set. if clusterTag == nil { return true } // If cluster name tag is set, then return true if it matches. if *clusterTag == clusterName { return true } } } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We deployed two Kubernetes clusters to the same Azure Resource Group. Created services of load balancer type on both clusters. One of the services never received a public IP. Upon investigation, we noticed that the first Kubernetes cluster deleted the Azure Public IP that was created for the service by the other cluster. Apparently, all public IPs in the resource group are retrieved: then public IPs which don't match this criteria: are deleted. This logic besides the name of the kubernetes service should also consider the K8s cluster where the public IP originated from to avoid stepping on each other's toes when multiple clusters share the same resource group. What you expected to happen: In case multiple clusters run in the same resource group a cluster should delete only resources created by itself. How to reproduce it (as minimally and precisely as possible): Create two Kubernetes clusters (1.14) with different cluster names to the same resource group than create kubernetes services of load balancer type in each cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for reporting the issue, this is the bug we should fix for sharing resource group. /sig azure /kind bug /assign


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "testing" ) func TestNetworkHostGetsPodNotFound(t *testing.T) { testKubelet := newTestKubelet(t, true) defer testKubelet.Cleanup() nh := networkHost{testKubelet.kubelet} actualPod, _ := nh.GetPodByName("", "") if actualPod != nil { t.Fatalf("Was expected nil, received %v instead", actualPod) } } func TestNetworkHostGetsKubeClient(t *testing.T) { testKubelet := newTestKubelet(t, true) defer testKubelet.Cleanup() nh := networkHost{testKubelet.kubelet} if nh.GetKubeClient() != testKubelet.fakeKubeClient { t.Fatalf("NetworkHost client does not match testKubelet's client") } } func TestNetworkHostGetsRuntime(t *testing.T) { testKubelet := newTestKubelet(t, true) defer testKubelet.Cleanup() nh := networkHost{testKubelet.kubelet} if nh.GetRuntime() != testKubelet.fakeRuntime { t.Fatalf("NetworkHost runtime does not match testKubelet's runtime") } } func TestNetworkHostSupportsLegacyFeatures(t *testing.T) { testKubelet := newTestKubelet(t, true) defer testKubelet.Cleanup() nh := networkHost{testKubelet.kubelet} if nh.SupportsLegacyFeatures() == false { t.Fatalf("SupportsLegacyFeatures should not be false") } } func TestNoOpHostGetsName(t *testing.T) { nh := NoOpLegacyHost{} pod, err := nh.GetPodByName("", "") if pod != nil && err != true { t.Fatalf("noOpLegacyHost getpodbyname expected to be nil and true") } } func TestNoOpHostGetsKubeClient(t *testing.T) { nh := NoOpLegacyHost{} if nh.GetKubeClient() != nil { t.Fatalf("noOpLegacyHost client expected to be nil") } } func TestNoOpHostGetsRuntime(t *testing.T) { nh := NoOpLegacyHost{} if nh.GetRuntime() != nil { t.Fatalf("noOpLegacyHost runtime expected to be nil") } } func TestNoOpHostSupportsLegacyFeatures(t *testing.T) { nh := NoOpLegacyHost{} if nh.SupportsLegacyFeatures() != false { t.Fatalf("noOpLegacyHost legacy features expected to be false") } } </ins>

This is a sub issue for
cc can you please assign this to me? Thanks.
, please create PR directly and cc to Tim for review.
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: imagePullPolicy: Never args: [ "--etcd-servers=http://localhost:2379" ] - name: etcd <del> image: gcr.io/etcd-development/etcd:v3.5.3 </del> <ins> image: gcr.io/etcd-development/etcd:v3.5.4 </ins>

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: claimAllocations []*controller.ClaimAllocation, selectedNode string), ) { <del> // pod1 will have only external-claim[2-3], it has to wait // for pod2 to consume resources from driver2 if claimAllocations[0].Claim.Name != "external-claim-other" { </del> <ins> if claimAllocations[0].Claim.Name == pod1claim2.Name { </ins> <-blockClaim.Done() } handler(ctx, claimAllocations, selectedNode) return } <ins> b.create(ctx, parameters1, parameters2, pod1claim1, pod1claim2, pod1) </ins> ginkgo.By("waiting for one claim from driver1 to be allocated") var nodeSelector *v1.NodeSelector gomega.Eventually(ctx, func(ctx context.Context) (int, error) {

What happened: Shortly after was merged, the "multiple nodes reallocation works" E2E DRA test started to fail randomly. Please provide links to example occurrences, if any: /sig node /kind bug needs to stay green. No response master n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: cp ${RELEASES_DIR}/kubernetes/server/kubernetes/server/bin/kubectl ${BINARY_DIR} fi <del> if [[ -f ${RELEASES_DIR}/docker ]]; then cp ${RELEASES_DIR}/docker ${BINARY_DIR}/node/bin </del> <ins> # docker if [[ -f ${RELEASES_DIR}/docker.tar.gz ]]; then tar xzf ${RELEASES_DIR}/docker.tar.gz -C ${RELEASES_DIR} cp ${RELEASES_DIR}/docker/docker* ${BINARY_DIR}/node/bin </ins> fi chmod -R +x ${BINARY_DIR}

Did a git clone and trying to build on centos. [root kubernetes]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root kubernetes]# make release +++ [0616 17:42:23] Verifying Prerequisites.... +++ [0616 17:42:57] Building Docker image kube-build:build-. +++ [0616 17:44:44] Running build command.... Usage of loopback devices is strongly discouraged for production use. Either use or use to suppress this warning. Go version: go version go1.6.2 linux/amd64 +++ [0616 17:45:01] Multiple platforms requested, but available 0G < threshold 11G, building platforms in serial +++ [0616 17:45:01] Building the toolchain targets: +++ [0616 17:45:04] Building go targets for linux/amd64: cmd/kube-dns cmd/kube-proxy cmd/kube-apiserver cmd/kube-controller-manager cmd/kubelet cmd/kubemark cmd/hyperkube federation/cmd/federation-apiserver federation/cmd/federation-controller-manager plugin/cmd/kube-scheduler go build /usr/local/go/pkg/tool/linuxamd64/compile: signal: killed !!! Error in 'go install "{goflags[ -ldflags "{nonstatics[ exited with status 1 Call stack: 1: kube::golang::buildbinariesforplatform(...) 2: kube::golang::buildbinaries(...) 3: hack/build- main(...) Exiting with status 1 !!! Error in '( kube::golang::setupenv; echo "Go version: (kube::golang::hostplatform); local goflags goldflags; eval "goflags=({KUBEGOLDFLAGS:-} do if [[ "${arg}" == "--usegobuild" ]]; then usegobuild=true; else if [[ "${arg}" == - ]]; then goflags+=("${arg}"); else targets+=("${arg}"); fi; fi; done; if [[ ${#targets[ -eq 0 ]]; then targets=("${KUBEALLTARGETS[ fi; local -a platforms=("{KUBEBUILDPLATFORMS[ if [[ ${#platforms[ -eq 0 ]]; then platforms=("${hostplatform}"); fi; local binaries;

binaries=({targets[ local parallel=false; if [[ ${#platforms[ -gt 1 ]]; then local gigs; gigs={gigs} -ge ${KUBEPARALLELBUILDMEMORY} ]]; then kube::log::status "Multiple platforms requested and available {KUBEPARALLELBUILDMEMORY}G, building platforms in parallel"; parallel=true; else kube::log::status "Multiple platforms requested, but available {KUBEPARALLELBUILDMEMORY}G, building platforms in serial"; parallel=false; fi; fi; kube::golang::buildkubetoolchain; if [[ "${parallel}" == "true" ]]; then kube::log::status "Building go targets for {targets[ local platform; for platform in "${platforms[ do ( kube::golang::setplatformenvs "{platform}: go build started"; kube::golang::buildbinariesforplatform {usegobuild:-}; kube::log::status "{platform////}.build" & done; local fails=0; for job in $(jobs -p); do wait ${job} let "fails+=1"; done; for platform in "${platforms[ do cat "/tmp//${platform////}.build"; done; exit ${fails}; else for platform in "${platforms[ do kube::log::status "Building go targets for {targets[ kube::golang::setplatformenvs "{platform} ${usegobuild:-}; done; fi )' exited with status 1 Call stack: 1: kube::golang::buildbinaries(.) 2: hack/build- main(.) Exiting with status 1 !!! Error in '" exited with status 1 Call stack: 1: kube::build::runbuildcommand(...) 2: main(...) Exiting with status 1 make:  [release] Error 1
Note that Centos downloads a bad "docker" binary that is an image with "404 Error". Centos seems abandonned by Kubernetes team... so I needed to make the entire operation with Ansible. And that is not simple at all :)
There was below info from your log: It seems like that the compiler was OOM killed due to memory pressure. AFAIK, consumes a lot of memory. I used to encounter when running it. We're going to close this issue.

Feel free to file a new issue if you fail to run at latest HEAD and you think there is a bug. Thanks!


Question: apiequality "k8s.io/apimachinery/pkg/api/equality" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/cli-runtime/pkg/genericclioptions" restclient "k8s.io/client-go/rest" "k8s.io/client-go/rest/fake" <ins> "k8s.io/client-go/tools/remotecommand" "k8s.io/kubectl/pkg/cmd/attach" </ins> "k8s.io/kubectl/pkg/cmd/delete" cmdtesting "k8s.io/kubectl/pkg/cmd/testing" cmdutil "k8s.io/kubectl/pkg/cmd/util"

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: } pod := podInfo.Pod <del> fwk.EventRecorder().Eventf(pod, nil, v1.EventTypeWarning, "FailedScheduling", "Scheduling", err.Error()) </del> <ins> msg := truncateMessage(err.Error()) fwk.EventRecorder().Eventf(pod, nil, v1.EventTypeWarning, "FailedScheduling", "Scheduling", msg) </ins> if err := updatePod(sched.client, pod, &v1.PodCondition{ Type:    v1.PodScheduled, Status:  v1.ConditionFalse,

What happened: When a pod can't be scheduled and the cardinality of taints on nodes is high, the Note field in the event can become bigger than the 1024 allowed and the event isn't sent. The pod stays in a state of without en event which makes the User Experience very bad What you expected to happen: I would expect the Kube-scheduler to be able to send the event no matter how many different taints are on nodes How to reproduce it (as minimally and precisely as possible): add many different taints to nodes and then trying to schedule something which can't be successfully scheduled (higher CPU/memory than any node has) Anything else we need to know?: In this case the length of the message was 1334 characters. Environment: Kubernetes version (use ): 1.18.9 Cloud provider or hardware configuration: bare metal Possible solutions: make event size configurable, it's now hardcoded to 1024 characters make scheduler note aware of that limit and work around I, if length 1024 then truncate the string (so at least something is shown) produce N events splitting the text (preferred one): change event Note to be to something like: (the whole complete information would still be present in the pod status field. for debugging purposes)
/sig scheduling
/triage accepted /cc
I could provide the fix in case we define which solution is best
This is a consequence of I think we can just drop some reasons when we are building the status message if the message is too large. This comes from here /help
/remove-help /assign
how many different taints did you have? I think in all cases we should truncate the message as well:
Another remidiation is to supress outoutping nil fields (such as GenerateName:"", SelfLink:"", UID:"", ResourceVersion:"", etc.) in the EventRecorder side.
The size limit we're hitting is specifically on the field
Completely agreed
good issue~
/assign
/unassign


Question: cmdutil.AddDryRunFlag(cmd) cmd.Flags().StringVar(&o.ClusterRole, "clusterrole", "", i18n.T("ClusterRole this RoleBinding should reference")) cmd.Flags().StringVar(&o.Role, "role", "", i18n.T("Role this RoleBinding should reference")) <del> cmd.Flags().StringArrayVar(&o.Users, "user", o.Users, "Usernames to bind to the role") cmd.Flags().StringArrayVar(&o.Groups, "group", o.Groups, "Groups to bind to the role") cmd.Flags().StringArrayVar(&o.ServiceAccounts, "serviceaccount", o.ServiceAccounts, "Service accounts to bind to the role, in the format <namespace>:<name>") </del> <ins> cmd.Flags().StringArrayVar(&o.Users, "user", o.Users, "Usernames to bind to the role. The flag can be repeated to add multiple users.") cmd.Flags().StringArrayVar(&o.Groups, "group", o.Groups, "Groups to bind to the role. The flag can be repeated to add multiple groups.") cmd.Flags().StringArrayVar(&o.ServiceAccounts, "serviceaccount", o.ServiceAccounts, "Service accounts to bind to the role, in the format <namespace>:<name>. The flag can be repeated to add multiple service accounts.") </ins> cmdutil.AddFieldManagerFlagVar(cmd, &o.FieldManager, "kubectl-create") return cmd }

I'm not sure whether this is the documentation being unclear, or a bug with kubectl itself but I suspect the former. Given this help page I expected to be able to use Instead I got the following error To get the same output as defining the flag twice aka and should give the same output, or the help page shouldn't list as the parameter type for --serviceaccount No response <details</details<detailsN/A </details<details</details<details</details<detailsarchitecture: amd64 bootID: -c6dc-40bb-9485- containerRuntimeVersion: containerd://1.5.5 kernelVersion: 5.4.0-88-generic kubeProxyVersion: v1.23.1 kubeletVersion: v1.23.1 </details<details</details
/sig cli
the flag can be specified multiple times, or (I think) as a comma-separated list. agree the notation is confusing... I think that's how cobra prints the default value (which is an empty list)
Hi I have tried and confirmed that it works with multiple flags: But for the comma-separated list, doesn't work. Kubectl version: v1.23.3 and k8s version v1.23.3
The same update for rolebinding: Should we write like this(same for clusterrolebinding):
Thanks for the investigation, another approach that might help is adding an example with the comma separated service accounts in the examples section. I do wonder if there's a clearer representation that could be output by cobra, to make it clear what the behaviour should be...?
Hi, I think adding the example is good so that users who reference this document get the proper information. but there is already mention of how to this flag with create command under the usage section: What do you suggest
in the specific flag help, we can mention the flag can be repeated to add multiple subjects
+1 I will update this. /assign
This sounds good to me, might be worth doing the same for other places that use the same behaviour?
Sure I will do that.
I have fixed it here: PTAL!
/triage accepted


Question: cinderSource := volume.Cinder pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", CinderDriverName, cinderSource.VolumeID), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: Individual pods are not intended to run multiple instances of the same application, in general. <ins> ## Alternatives considered </ins> Why not just run multiple programs in a single Docker container? 1. Transparency. Making the containers within the pod visible to the infrastructure enables the infrastructure to provide services to those containers, such as process management and resource monitoring. This facilitates a number of conveniences for users. <del> 2. Decoupling software dependencies. The individual containers may be rebuilt and redeployed independently. Kubernetes may even support live updates of individual containers someday. </del>  No newline at end of file <ins> 2. Decoupling software dependencies. The individual containers may be rebuilt and redeployed independently. Kubernetes may even support live updates of individual containers someday. 3. Ease of use. Users don't need to run their own process managers, worry about signal and exit-code propagation, etc. 4. Efficiency. Because the infrastructure takes on more responsibility, containers can be lighterweight. Why not support affinity-based co-scheduling of containers? That approach would provide co-location, but would not provide most of the benefits of pods, such as resource sharing, IPC, guaranteed fate sharing, and simplified management. </ins>

Currently replication controllers define an exact value. Are there disadvantages to allowing the replication controller to define more of a policy - at least X, at most X, exactly X? Since replication controllers aren't really aware of overlapping sets, I guess you can't really use that to guide allocation between two pools because they would just end up fighting unproductively
What is the use case? This seems sort of like auto-scaling policy. I'd prefer to put those objects in the auto-scaler.
It sounds like you want a canary-controller that scales the old version down while scaling the new one up. Combine with a service that spreads traffic across both versions and this should Just Work (tm).
I'm thinking about this from the mechanics of doing a deployment by composing replication controllers. With controllers as they are today: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo,deployment=2 x0 alter controller 2 to be x1 (or up by the batch size) alter controller 1 to be x4 (or down by the batch size) ... repeat until done delete controller 1 The thought around atLeast was about explicitly representing the invariant you want during an available deployment: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo, atLeast=5 set controller to be atMost=5 delete a pod from the name=foo,deployment=1 query ... continue until 0 pods in that query delete controller 1 update controller 2 to change the label query to name=foo,deployment=2 I'm not sure that's better.... just the for loop for deletes seems more reasonable. Not a huge deal, just trying to work through how we'd model common deploy cases for end users.
Overlapping replicationController sets: I think we should validate that this isn't possible upon creation of a replicationController. We can do this by determining whether the label selector may overlap with any others, though to do this analytically without looking at the combinations of labels attached to pods it requires that all replicationController label selectors have at least one key in common, with a different value than all other replicationControllers. Policy: As mentioned, we intended the policy to go in a separate auto-scaling policy entity, in a higher-level layer / separate component.

Deployment: As pointed out, it is possible to create a new replicationController and gradually scale it up while scaling the original down. Or, one could do the Asgard trick of creating a new replicationController and then shift traffic to it. It's possible to add/remove instances from a service by tweaking their labels. For instance one could toggle the value of an in_service=true/false label. The update demo changes the template of a single replication controller and then kills instances one by one until they all get replaced. I don't like that due to the risk of unpredictable replacements and complexity involved with rollbacks. Splitting out the template ( ) would address the rollback issue, but not the unpredictability. It seems to me that the atMost approach suffers from the same risk. That said, I see what you mean that atLeast/atMost are not auto-scaling policies, but are invariants to maintain. I could imagine that flexibility could be useful. For example, a worker pool manager could use that to opportunistically request resources, where some pod requests may fail. Or, perhaps in the case that we need to temporarily create a new pod for some reason (CRIU-based migration?) but we don't want an old one to be killed. So, I like the idea, just not the use case. :-) FWIW, exactly X could be implemented by combining atLeast and atMost.
Agree that without a kill policy on the replication controller the uses cases described don't work. Is a kill policy something you've discussed for the replication controller (oldest, ones that don't match my template first, ones on most overloaded systems)?
I agree that kill policy would be useful and all of the policies you mention are reasonable in particular scenarios. However, I see this as a rich, unbounded policy space, much like scheduling. IIRC, Joe and/or Brendan proposed kill policies early on and I asked for them to be removed. Instead, I propose that we leave the default policy entirely up to the system, much like scheduling, so we can take as many factors into account as we like, and otherwise I'd like to enable the user to implement an arbitrary policy using event hooks . For instance, I could imagine POSTing to a URL whenever a pod needed to be killed.

The user's hook would have N seconds to respond (perhaps configurable), after which the replication controller would kill its default choice.
Thanks for the clarification - I think I'll close this for now until the API stabilizes, and then add some of this context to the doc.
I suggest re-opening this issue, but change the title to "add XXX context to doc YYY", so that we can reference in the PR for posterity.


Question: // Metrics to track ephemeral container usage by this kubelet ManagedEphemeralContainersKey = "managed_ephemeral_containers" <ins> // Metrics to track the CPU manager behavior CPUManagerPinningRequestsTotalKey = "cpu_manager_pinning_requests_total" CPUManagerPinningErrorsTotalKey   = "cpu_manager_pinning_errors_total" </ins> // Values used in metric labels Container          = "container" InitContainer      = "init_container"

Add to enable observability of the cpumanager. This is required to but it's also useful in general. To the best of my knowledge, it's very hard to observe the behavior of cpumanager without peeking into the pod and/or use the podresources interface (kubelet = 1.24). But even for the latter, no well known components expose metrics consuming this information. I believe anyway kubelet some expose some basic metrics about cpumanager behaviour, while deferring more fine grained metrics to external components (NFD, node_exporter, is fine)
/sig node /assign
/triage accepted /priority important-soon (as this is important for graduation for CPU Manager to GA)


Question: Selector  string Namespace string <del> Describer func(*meta.RESTMapping) (printers.Describer, error) Builder   *resource.Builder </del> <ins> Describer  func(*meta.RESTMapping) (printers.Describer, error) NewBuilder func() *resource.Builder </ins> BuilderArgs []string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I run and I got Running with shows What you expected to happen: How to reproduce it (as minimally and precisely as possible): With Kubernetes master branch and in different terminal on the same machine Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM via libvirtd OS (e.g. from /etc/os-release): Fedora 27 with golang-1.10.1-1.fc29.x86_64 from rawhide Kernel (e.g. ):Install tools: ?? Others:
/sig cli
/assign
/assign


Question: github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e/go.mod h1:+SR5DhBJrl6ZM7CoCKvpw5BKroDKQ+PJqOg65H/2ktk= github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 h1:J9gO8RJCAFlln1jsvRba/CWVUnMHwObklfxxjErl1uk= github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936/go.mod h1:ZjcWmFBXmLKZu9Nxj3WKYEafiSqer2rnvPr0en9UNpI= <del> github.com/vmware/govmomi v0.20.1 h1:7b/SeTUB3tER8ZLGLLLH3xcnB2xeuLULXmfPFqPSRZA= github.com/vmware/govmomi v0.20.1/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </del> <ins> github.com/vmware/govmomi v0.20.3 h1:gpw/0Ku+6RgF3jsi7fnCLmlcikBHfKBCUcu1qgc16OU= github.com/vmware/govmomi v0.20.3/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU= </ins> github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8= github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 h1:j2hhcujLRHAg872RWAV5yaUrEjHEObwDv3aImCaNLek=

Expectation- We should be able to create PVCs via a storageclass with storagePolicyName defined while using a singed ca cert for secure communications with vCenter. Bug- The PVC created with a storageclass with storagePolicyName defined fails with a error when PVCs created with a storageclass without the storagPolicyName work against the same vCenter and datastore. Scenario: Utilizing the vSphere in-tree plugin on kubernetes 1.12.7. Utilizing secure communication with vSphere (signed ca cert imported into the cluster) We are able to create PVCs with the following storage class: However, if we include a storagePolicyName we receive a failure: Here is the yaml for our sc1 storage class: /sig vmware
I think we are having issue with creating PBM client using certificate set on the vCenter Client. We need to create a PBM client to get list of compatible datastore for given storage policy. I think call is failing at - Can you please provide controller-manager logs for further debugging. Make sure log level is set to 4.
Sorry for the delayed response. I'm not confident I got the log level correct, however this is what I find:
I believe I have log level 4 now set:
For comparison, here is a PVC created w/o storagePolicyName on the same datastore which succeeds:
thanks for posting logs. Now this is confirmed, issue is happening while creating PBM client when we use certificate based authentication. One more thing I want to confirm regarding the certificate, are you using solution user as mentioned here -
Can you also provide value of in the file? Is it set to true or false / 1 or 0 ? Also confirm version of vCenter you are using.
is unset, as for the solution user I'd have to leave that to to expand on.
I don't believe we used any solution user because we simply used "" as the default user during installation. please correct me if I'm wrong.
How is the certificate and private key retrieved for the user . I am successfully able to connect to the PBM using vpxd certificte and key. I ssh into the vCenter and executed following commands to retrieve certificate and private key for vpxd. with vpxd cert and private key, I do not see this issue. In addition I also set while making connection.

Can you try setting Insecure to false? That's the flag setting that triggers the cert error.
with , I see the exact same error - , while creating VC client.
Isn't that because it's self-signed? In the case of the issue in this ticket the cert is signed by a trusted CA /cc to confirm.
The cert is self-signed and its CA isn't trusted.
I used vpxd certificate and issuer is I require to set .
Is there any way to make it work if insecure=false?
- then I’d say that’s expected? If the root CA isn’t trusted, then it is insecure?
well, while that's expected, I don't know if there's a method to add the root CA to a trusted list.
Correct me if I’m wrong but isn’t the trusted list the root CA store of the K8S node’s OS in this case? I’d imagine whatever CA Root signed the cert would need to be to the OS of choice’s CA store.
you are correct, we need to download trusted root certificate and add them in the store on node vm. For now on my mac, I have downloaded trusted root certificate from the vCenter -and them in the keychain and flagged as "Always Trust". ! After this, I am able to connect to VC client with or without specifying this flag at all using VPXD certificate and private key. So certificate needs to be to the CA trust store on the master node VM. after adding trusted root certificate from vCenter to the trust store, you should be able to make a connection using .
After importing the root cert by placing it in /etc/ssl/certs on the master node, I'm able to provision a PVC while specifying a storage policy. Thanks for all the help.
Can you close this issue and work with KK to get this automated in the product?
It turns out that importing the root cert into the host was just a workaround, and the actual bug was fixed in That fix is included in govmomi v0.21.0.


Question: and current_state['details']['mtu'] == mtu and desired_network in current_state['details']['networks'] and current_state['details']['up'] <del> and current_state['iptables_rule_exists']): </del> <ins> and current_state['iptables_rule_1_exists'] and current_state['iptables_rule_2_exists']): </ins> ret['result'] = True ret['comment'] = 'System already in the correct state' return ret

Spin up a cluster. Create a container on each of two machines. Try to ping each other's IPs. No love. It seems we need an extra iptables rule. For example, for a minion with the 10.244.2.0/24 subnet assigned, we need: iptables -t nat -A POSTROUTING -s 10.244.2.0/24 ! -d 10.244.2.0/24 -j MASQUERADE
Hrm, I can't repro this. I tried: ping 10.244.2.1 from a different kubernetes VM and it worked. (or do you mean from inside a container?)
From within a container to another container
repro'ed it for me, too.
Is this just ping? Or is this any network traffic? , Tim Hockin wrote:
Any traffic - the kernel is not masquerading it because our iptables magic doesn't match packets coming from inside a container on a veth. I saw this while trying to do another round of perf tests.


Question: ignorable         bool gotBind           bool isPrioritizer     bool <ins> errBind           bool </ins> } func (f *fakeExtender) Name() string {

I use extenders to add webhook for default-scheduler: And set to ignore scheuler error when scheduler extender is not working. But I got an error: . I suppose work fine in predicate path, but not work fine in bind path. The Ignorable logic in kube-scheduler: 637: and, 905: the is not judge in . Schedule should not error when set . extenders to kube-scheduler config(which has filterVerb and bindVerb and ignorable flag), example: and add config flag in kube-scheduler pod start up command. a scheduler-extender() and then stop it. a pod and you will see the schedule error. No response <details</details<detailsLocal environment </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Maybe this is not a bug but as designed, because bind is more critical than filter, what would you do in bind? I have no history context, cc
Hi, Kerthcet, in my scene, I use extender to bind IP(from my own network system) to my pod's annotaion. I'm aware of the critical of binding path, but, what If the extender is broken, how to skip it and continue? The scheduler extender design has supported ignorable, but only work for filter path make no sense for extender both has filter and bind. Maybe we can add an option to allow user to chose what can be ignored. Thanks for your reply anyway~
Reiterating the mentions to trigger a notification: <detailsIn response to : Maybe this is not a bug but as designed, because bind is more critical than filter, what would you do in bind? I have no history context, cc I'm aware of the critical of binding path, but, what If the extender is broken, how to skip it and continue?

The scheduler extender design has supported ignorable, but only work for filter path make no scence for extender both has filter and bind. Maybe we can add an option to allow user to chose what can be ignored. thanks for your reply anyway~ Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So even the extender bind failed, will not impact your pod scheduling, right?
Yup
Right, if the field is there, we should support it. It's up to the cluster administrator to judge whether it makes sense to ignore a failed extender during bind.


Question: options := []string{"bind"} <del> if *volumeSource.ReadOnly { </del> <ins> if volumeSource.ReadOnly != nil && *volumeSource.ReadOnly { </ins> options = append(options, "ro") }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!


Question: package storage import ( <del> "time" </del> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

/assign The test flakes presubmit testing in PRs six times per week The site can focus on just this test: Which shows that 83 out of 85 errors match this pattern:
Previous attempt to address this flake:
CC
Took a quick look at a failed test: https://k8s- The indicates: . or and can you help investigate.
CC
The failure in the GKE multizone test suite is probably due to the GKE issue I fixed last week.
We'll dig into this and see what's up.
Expectations: This work should interrupt and take priority over other activity while it is assigned to you. Expectation for sig leads is to identify someone to fix the issue, them and yourself. Expectation for assignees are to investigate the issue and either a) resolve it or b) it to a more appropriate person and b) yourself.
I repeatedly ran this test on my e2e cluster since 7pm last night, and it has not failed yet. Seems like it fails more frequently in the PR env instead of the CI env, so may need to look into differences between the two. The test is failing during the deletion of the namespace. It seems like there are still some resources not getting deleted. I'm trying to dig deeper in the logs to see if I can find what resources are not getting cleaned up.
Here's one failed : From the API server initial namespace deletion request from the e2e test: I0427 23:23:29. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (4.744775ms) 200 [[ (linux/amd64) kubernetes/] 104.197.101.155:] controller triggers deletion on all the objects in the namespace: I0427 23:24:37. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (91.176059ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] I0427 23:24:37.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (7.608267ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0427 23:26:30. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (87.184389ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serv iceaccount:kube-system:namespace-controller] [::1]:] I0427 23:26:30. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (21.517822ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccoun t:kube-system:namespace-controller] [::1]:] etc ... deletion finished I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (16.58598ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] 2E test cleanup tries to delete it again: I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (9.104808ms) 404 [[ (linux/amd64) kubernetes/] 104.197.101.155:] <= This is the e2e test failure cleanup Total time to delete 3 mins Here's a successful : : deletion request I0428 00:45:14. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (11.208829ms) 200 [[ (linux/amd64) kubernetes/] 104.197.153.246:] starts object deletions I0428 00:45:46.

5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (33.104152ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:servi ceaccount:kube-system:namespace-controller] [::1]:] I0428 00:45:46. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (3.447937ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:k ube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0428 00:46:17. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (10.679029ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount: kube-system:namespace-controller] [::1]:] I0428 00:46:20. 5 ] DELETE /apis/extensions/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (6.130617ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system: serviceaccount:kube-system:namespace-controller] [::1]:] etc ... deletion is finished I0428 00:46:21. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (21.10359ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:na mespace-controller] [::1]:] 2E test cleanup I0428 00:46:32.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (1.436939ms) 404 [[ (linux/amd64) kubernetes/] 104.197.153.246:] Total time to delete ~1 minute Comparing the bad and good runs, these were some supicious areas: the bad run, the period between 1) and 2) was 1 minute, whereas in the good run it was 30 seconds the bad run, the period between 2) and 3) was ~2 minutes, whereas in the good run it was ~20 seconds. The slowness in the bad run seems systematic, I did not find any one object that was taking a particularly long time to delete, all of them seemed to be taking longer compared to the good run. I'm not sure where to dig down further to find the cause of the general slowness. I don't know if this is expected, and we should just increase the timeout? The test framework seems to have a default timeout of 5 minutes, whereas this volume test timeout is only 3 minutes.
Great analysis! Since no singular event stands out as taking too much time how about, in the name of expediency, just changing the timeout to 5m and see how the tests behave?
The delay of ns deleteion might related to this PR and . is still open.
Those two issues do not seem to impact the deletion of the namespace itself, so I don't think fixing those will solve this issue.
But I think delaying the deletion of a pod will affect the time to delete the namespace.


Question: mockClient := fake.NewSimpleClientset(testInput.testNamespace) if containsVersion(versions, "extensions/v1beta1") { resources := []unversioned.APIResource{} <del> for _, resource := range []string{"daemonsets", "deployments", "jobs", "horizontalpodautoscalers", "ingresses"} { </del> <ins> for _, resource := range []string{"daemonsets", "deployments", "replicasets", "jobs", "horizontalpodautoscalers", "ingresses"} { </ins> resources = append(resources, unversioned.APIResource{Name: resource}) } mockClient.Resources = map[string]*unversioned.APIResourceList{

The delete command for a replicaset hangs forever. This then manifests as phantom RSs when I delete and recreate a namespace.
- delete is the gift that keeps on giving
cc
Had you already deleted the corresponding Deployment?
I am still seeing this problem. Deployments and replica sets are sticking.
From what I can tell, calling DELETE_COLLECTION on Deployments is returning: And since the controller assumes not found errors are equivalent to already deleted, it thinks the content is deleted. When I manually inspect, I still see the deployment and replica set laying around... //cc
holding on this while i debug some more...
This was a false alarm, apologies.


Question: pkg/apis/core/v1 pkg/apis/core/v1/helper pkg/apis/core/v1/helper/qos <del> pkg/apis/core/v1/validation </del> pkg/apis/core/validation pkg/apis/discovery/v1alpha1 pkg/apis/discovery/v1beta1

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if serr, ok := err.(*errors.StatusError); ok && serr.Status().Reason == metav1.StatusReasonNotFound && serr.ErrStatus.Details.Name == "" { return nil, "", fmt.Errorf("ephemeral containers are disabled for this cluster (error from server: %q).", err) } <ins> // The Kind used for the /ephemeralcontainers subresource changed in 1.22. When presented with an unexpected // Kind the api server will respond with a not-registered error. When this happens we can optimistically try // using the old API. if runtime.IsNotRegisteredError(err) { klog.V(1).Infof("Falling back to legacy API because server returned error: %v", err) return o.debugByEphemeralContainerLegacy(ctx, pod, debugContainer) } </ins> return nil, "", err } return result, debugContainer.Name, nil } <ins> // debugByEphemeralContainerLegacy adds debugContainer as an ephemeral container using the pre-1.22 /ephemeralcontainers API // This may be removed when we no longer wish to support releases prior to 1.22. func (o *DebugOptions) debugByEphemeralContainerLegacy(ctx context.Context, pod *corev1.Pod, debugContainer *corev1.EphemeralContainer) (*corev1.Pod, string, error) { // We no longer have the v1.EphemeralContainers Kind since it was removed in 1.22, but // we can present a JSON 6902 patch that the api server will apply. patch, err := json.Marshal([]map[string]interface{}{{ "op":    "add", "path":  "/ephemeralContainers/-", "value": debugContainer, }}) if err != nil { return nil, "", fmt.Errorf("error creating JSON 6902 patch for old /ephemeralcontainers API: %s", err) } result := o.podClient.RESTClient().Patch(types.JSONPatchType). Namespace(pod.Namespace). Resource("pods"). Name(pod.Name). SubResource("ephemeralcontainers"). Body(patch). Do(ctx) if err := result.Error(); err != nil { return nil, "", err } newPod, err := o.podClient.Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{}) if err != nil { return nil, "", err } return newPod, debugContainer.Name, nil } </ins> // debugByCopy runs a copy of the target Pod with a debug container added or an original container modified func (o *DebugOptions) debugByCopy(ctx context.Context, pod *corev1.Pod) (*corev1.Pod, string, error) { copied, dc, err := o.generatePodCopyWithDebugContainer(pod)

changed the Kind used with the subresource from to . This change will break old clients. (The ephemeral containers API is still in alpha, so this is acceptable.) Currently, compatibility is broken in both directions: versions prior to 1.22 are incompatible with server version 1.22 version 1.22 is incompatible with server versions prior to 1.22 Remediating the first requires complex server-side backwards compatibility logic which is not worth implementing for an alpha API. Remediating the second for should be much easier. The struct hasn't changed, only the path in the object. We could detect the kind returned by the server and change the path used in the JSON patch (or even just naively try the previous path if the server returns an error). This behavior would be removed after the old server version falls out of the support window. This isn't strictly needed, but it would improve UX for what I expect is a common scenario: using the latest kubectl with older servers. /sig cli /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about adding this backwards compatibility behavior to kubectl?
Are we able to detect this and fail gracefully client side with a warning to users to upgrade their client? Agree that this fix is the important one. I wonder if a commit for backwards compatibility is something we can cherry pick to other release branches? Seeing as earlier branches will have both structs it might even be cleaner to do version conditional in kubectl there? is that something we've done before?
Just to make in my head a matrix of effects/noneffects, please confirm for me if these are right: 1.22 will have the compatibility logic baked in. So it will be able to talk to older servers. -picking that logic to older branches would not alter in any way how pre-1.22 kubectls interact with old (pre 1.22) servers.

cherry-pick would only add the needed logic to allow old kubectls to talk to 1.22 and later servers Are these assumptions correct?
Correct. I haven't prototyped it, but this should be possible. We could trigger the logic on a server error that indicates it received an unexpected Kind. That being said, I don't think it's worth cherry-picking forward compatibility into older versions of kubectl. This is for an alpha feature that must be explicitly enabled with the feature flag. It's not unreasonable to require people experimenting with an alpha feature to use a kubectl version = server version. There's far more risk than benefit. It would be more of a burden to keep around old versions of kubectl for older clusters, so I thought adding backwards compatibility in 1.22 would be nice, even if it's not strictly required. I agree it would be nice if we could give users of pre-1.22 kubectl a better error message, but I at least have kubernetes/website out to indicate a minimum kubectl version requirement of 1.22.
I'd like to take up this issue. /assign
How's it going? I'd like to make sure this makes it into 1.22.
Hi , I am able to reproduce this issue. I am looking for some guidance on how to go about fixing this issue. You have mentioned that "detect the kind returned by the server and change the path used in the JSON patch ", where is the kind being returned by the server and what JSON patch are you referring to?
Hi given that time zones are difficult and the code freeze is near, I think it'll be most efficient to communicate with examples. I'll prepare a PR with what I have in mind.


Question: "//pkg/util/metrics:go_default_library", "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", <del> "//vendor/k8s.io/apimachinery/pkg/labels:go_default_library", </del> "//vendor/k8s.io/apimachinery/pkg/util/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library",

For reasons that are not entirely clear to me, and as spotted by in , there are two functions for selecting nodes in includeNodeFromNodeList & getNodeConditionPredicate. I think includeNodeFromNodeList should use the logic in getNodeConditionPredicate
/sig network


Question: "github.com/emicklei/go-restful/swagger" "k8s.io/kubernetes/pkg/api" <del> "k8s.io/kubernetes/pkg/api/latest" </del> "k8s.io/kubernetes/pkg/api/unversioned" "k8s.io/kubernetes/pkg/version" )

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


Question: <ins> apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: default annotations: storageclass.beta.kubernetes.io/is-default-class: "true" labels: kubernetes.io/cluster-service: "true" provisioner: kubernetes.io/vsphere-volume parameters: diskformat: thin </ins>

We'd like to deprecate alpha provisioning in 1.6. In order to make current deployments working after upgrade to 1.6 we need to deploy a default StorageClass as part of installation/upgrade on vSphere so it will automatically provision volumes for PVCs marked with annotation - the annotation will be ignored and default storage class must provision the volume instead. On GCE and AWS, we use addon manager to install the default storage class, see Note that some extra code was needed in AWS and GCE installers to actually put the storage class into addons directory during installation, e.g. Please make sure that: A default StorageClass is deployed during "default" installation. It would be nice if admin could turn this off during installation (e.g. as on GCE) If any additional manual steps are needed to deploy the default storage class make sure they're documented in appropriate installation guide.
cc
thanks will look into this. CC
Documentation update for cloud provider


Question: in  string out string }{ <del> {`aoeu`, `aoeu ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `}, {`aoeu [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() `, `aoeu ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `}, {`aoeu ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `, `aoeu ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `}, {`aoeu [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `, `aoeu ` + beginMungeTag("GENERATED_ANALYTICS") + ` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]() ` + endMungeTag("GENERATED_ANALYTICS") + ` `}, </del> <ins> { "aoeu", "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, { "aoeu" + "n" + "n" + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()", "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, { "aoeu" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n", "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, { "aoeu" + "n" + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n", "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, { "prefix" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n" + "suffix", "prefix" + "n" + "suffix" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, { "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n", "aoeu" + "n" + "n" + "n" + beginMungeTag("GENERATED_ANALYTICS") + "n" + "[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/path/to/file-name.md?pixel)]()" + "n" + endMungeTag("GENERATED_ANALYTICS") + "n"}, </ins> } for _, c := range cases { out, err := checkAnalytics("path/to/file-name.md", []byte(c.in))

If you try mimicking the release flow for a release on right now, it's busted: If I generate a dummy commit out of that failure, then run to see what gets generated, it's attempting to actually add the unversioned warnings back in: I was trying to write up a PR to enact something for , but this blocks the release piece for that. Which is fine, we have some time.
cc
Running gendocs should add the unversioned warnings, unless the doc has <!-- TAG IS_VERSIONED --. Do you know if mark-new-version that tag before failing?
Ah, I think I see what might be happening. This is the post- version of Notice the newline fun.
Sorry which newline are you talking about? :)
My PR fixes that
Ah, yes, cool. It would help if you flagged functional changes along more loudly. :)
Yea, sorry about that. Have updated the PR description to highlight each fix.


Question: // Check whether availability zone is specified. if disk.Zones == nil || len(*disk.Zones) == 0 { klog.V(4).Infof("Azure disk %q is not zoned", diskName) <del> return nil, nil </del> <ins> return labels, nil </ins> } zones := *disk.Zones

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Overview A persistent volume claim / storage class configured in a certain way can cause all the Kube API servers on an ARO cluster to crashloop. We’ve seen this issue on a production cluster (ICM ). In development we have proven one vector by which the cluster could have got into this state but it doesn’t seem that likely. We wonder if there are other vectors. We don’t currently have a good way to rescue a cluster that gets into this state because our service accesses are predicated on API server availability. Request We would like help: agree and expedite an appropriate fix in order to protect all other production clusters. suggestions on how we might fix the broken production cluster. Causes certain cases upon PV create, the persistentvolume admission controller attempts to retrieve the managed disk object via the Azure API, but in ARO the relevant client object is a nil pointer. The pointer is dereferenced and the kube API server panics. PVC controller repeatedly attempts to create the PV object, which DoSes the kube API server. client object in (1) is nil because of ARO’s cloud provider authentication architecture, which differs from that of OCP. More information below. Technical detail For a panic stack trace and reproducer, see Creation of an Azure Disk persistent volume is attempted The persistentvolume admission controller runs, (l persistentVolumeLabel) Admit(), in In our possible reproducer, the admission controller does not return early in (l persistentVolumeLabel) findVolumeLabels(), because the zone labels are not set.

A cloud provider is (incorrectly) instantiated via: getAzurePVLabeler(), cloudprovider.GetCloudProvider(), NewCloud(), (az Cloud) InitializeCloudFromConfig, We see the following loglines: Azure cloud provider is starting without credentials Azure cloudprovider using try backoff: retries=%d, exponent=%f, duration=%d, jitter=%f We believe that an early return occurs at , before c.DisksClient is initialised, because servicePrincipalToken is nil Later, a nil pointer dereference of c.DisksClient occurs in (c Cloud) GetAzureDiskLabels, OCP 4.3 cloud provider authentication architecture ================================================== The installer lays down an Azure managed identity (MSI) resource, grants it Contributor access to cluster resource group and to any pre-existing virtual network (VNET). In the context of the Kube API server, the persistentvolume admission controller uses MSI to authenticate to Azure for its API calls. ARO 4.3 cloud provider authentication architecture ================================================== ARO can’t follow OCP 4.3’s architecture because of the following chicken-and-egg problem: As an Azure service, ARO is not permitted to add role assignments to the customer’s pre-existing virtual network - the customer must explicitly do this. No cluster managed identity can exist before the customer runs (by definition), so there is nothing for the customer to grant the role assignment to. As a result, ARO does not use a managed identity at all, and instead uses a user-created cluster service principal. ARO carries a patch to the installer which creates a kube-config/azure-cloud-provider Secret during cluster install. The controller manager merges the cloud config with this Secret at start-up time (see ), however this code path does not run in the API server. In ARO, any calls to the Azure API from the API server are probably broken. Prior to GA, ARO wrote the cloud config secret raw into the cloud config. Reverting to this behaviour would work around this issue, but is clearly suboptimal. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
/assign


Question: } return pathRecorderMux } <ins> // WithPlugin creates an Option based on plugin name and factory. func WithPlugin(name string, factory framework.PluginFactory) Option { return func(registry framework.Registry) error { return registry.Register(name, factory) } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: func (proxier *Proxier) addServiceOnUnusedPort(service string) (string, error) { unusedPortLock.Lock() defer unusedPortLock.Unlock() <del> l, err := net.Listen("tcp", ":0") </del> <ins> l, err := net.Listen("tcp", net.JoinHostPort(proxier.address, "0")) </ins> if err != nil { return "", err }

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote:


Question: dest.Components.Schemas = map[string]*spec.Schema{} } if _, exists := dest.Components.Schemas[k]; exists { <del> klog.Warningf("Should not happen: OpenAPI V3 merge schema conflict on %s", k) </del> <ins> if strings.HasPrefix(k, metadataGV) || strings.HasPrefix(k, autoscalingGV) { continue } return fmt.Errorf("OpenAPI V3 merge schema conflict on %s", k) </ins> } dest.Components.Schemas[k] = v }

Related to Seeing the following errors in kube-apiserver logs when CRDs are applied: No errors in logs Apply any CRD manifest No response <details</details<details</details<details</details<detailsRancher RKE2 </details<detailscontainerd 1.6.4 </details<detailscanal </details
/sig api-machinery
/assign
Did this cause any specific trouble to your cluster I'm curious what the impact/severity of this is.
Everything seems to work fine, but there's a bunch of log spew every time a CRD manifest is loaded. In K3s we mix all the control-plane component logs together in a single pair of stdout/stderr streams so it's quite noisy, compared to previous releases. The fact that these come out at WARN level with a scary prefix makes them look pretty serious, and I expect we will get a fair bit of users asking about it once we release.
/triage accepted


Question: defer framework.RemoveLabelOffNode(cs, nodeName, k) // make the nodes have balanced cpu,mem usage ratio <del> createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) </del> <ins> err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) framework.ExpectNoError(err) </ins> By("Trying to relaunch the pod, now with labels.") labelPodName := "pod-with-node-affinity" pod := createPausePod(f, pausePodConfig{

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: PodWorkerStartLatencyKey      = "pod_worker_start_latency_microseconds" PLEGRelistLatencyKey          = "pleg_relist_latency_microseconds" PLEGRelistIntervalKey         = "pleg_relist_interval_microseconds" <ins> EvictionStatsAgeKey           = "eviction_stats_age_microseconds" </ins> // Metrics keys of remote runtime operations RuntimeOperationsKey        = "runtime_operations" RuntimeOperationsLatencyKey = "runtime_operations_latency_microseconds"

We should keep track of how stale stats used for evictions are. This will help track improvements in eviction latency, and help uncover regressions. cc If someone could assign this to me, I would appreciate that.
closing via


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", <del> "//vendor/k8s.io/klog:go_default_library", </del> ], )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Logging getting points for ALL priorities set in the default scheduler policy Why is this needed: We have Kubernetes v1.14 with a problem of planning pods for three nodes. The problem is that the scheduler unevenly distributes the pods across the nodes. For example, the CPU on two nodes Node02 and Node03 are 50% loaded, while the CPU on the third Node01 is not loaded at all. While that the pods continue to be planned on Node02 and Node03, but they are not planned on Node01 at all. Scheduler logs (maximum logging level) with uniform planning: and uneven planning: As you can see, the sum of points for the priorities BalancedResourceAllocation, LeastResourceAllocation, SelectorSpreadPriority, TaintTolerationPriority is the same in both cases, but the final result is different. There is no way to understand for what specific policy priorities Node01 did not get 3 points, for to be on a par with the others, because the set of points of some priorities is not logged.
/sig scheduling
I think maybe we missed some logs which can reflect the situation of giving the score. WDYT? If you confirmed, I want to handle this issue.
Thank you for the answer! Here is the full scheduling log (logging level 10): maybe I’m really missing something, but I don’t see information about other priorities here.
Missing scores may be due to some priority functions are using non-map-reduce style, so they didn't hit the printing logic. We can add a logging function there to help you better debugging it. However, the code gets refactored in recent releases, and 1.14 is out of our supporting cycles (only 1.15, ,1.16, 1.17). Can you upgrade to a newer version, or I can provide a code snippet for you to compile on your local 1.14 branch?
Thank you, but I already solved my problem by creating an additional scheduler with a custom scheduling policies for my cronjob. However, it would be cool if you the logging function in the supported versions :)
SG. I created to address this.
I'm running v1.23.8, and I've tried and I'm still not seeing any of the scoring output you in Am I doing something wrong?

Shouldn't this be available in v1.23?
is it possible there is only 1 node candidate after filtering? In that case, the scoring logic will be bypassed as it's meaningless to score for 1 node. If it's not the case, could you check the beginning of the scheduler log, to ensure is configured properly?


Question: genericfeatures.ValidatingAdmissionPolicy: {Default: false, PreRelease: featuregate.Beta}, <del> genericfeatures.CustomResourceValidationExpressions: {Default: true, PreRelease: featuregate.Beta}, </del> <ins> genericfeatures.CustomResourceValidationExpressions: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 </ins> genericfeatures.OpenAPIEnums: {Default: true, PreRelease: featuregate.Beta},

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc