Question: <ins> --- title: 服务账号 description: > 了解 Kubernetes 中的 ServiceAccount 对象。 content_type: concept weight: 10 --- <!-- title: Service Accounts description: > Learn about ServiceAccount objects in Kubernetes. content_type: concept weight: 10 --> <!-- overview --> <!-- This page introduces the ServiceAccount object in Kubernetes, providing information about how service accounts work, use cases, limitations, alternatives, and links to resources for additional guidance. --> 本页介绍 Kubernetes 中的 ServiceAccount 对象， 讲述服务账号的工作原理、使用场景、限制、替代方案，还提供了一些资源链接方便查阅更多指导信息。 <!-- body --> <!-- ## What are service accounts? {#what-are-service-accounts} --> ## 什么是服务账号？  {#what-are-service-accounts} <!-- A service account is a type of non-human account that, in Kubernetes, provides a distinct identity in a Kubernetes cluster. Application Pods, system components, and entities inside and outside the cluster can use a specific ServiceAccount's credentials to identify as that ServiceAccount. This identity is useful in various situations, including authenticating to the API server or implementing identity-based security policies. --> 服务账号是在 Kubernetes 中一种用于非人类用户的账号，在 Kubernetes 集群中提供不同的身份标识。 应用 Pod、系统组件以及集群内外的实体可以使用特定 ServiceAccount 的凭据来将自己标识为该 ServiceAccount。 这种身份可用于许多场景，包括向 API 服务器进行身份认证或实现基于身份的安全策略。 <!-- Service accounts exist as ServiceAccount objects in the API server. Service accounts have the following properties: --> 服务账号以 ServiceAccount 对象的形式存在于 API 服务器中。服务账号具有以下属性： <!-- * **Namespaced:** Each service account is bound to a Kubernetes {{<glossary_tooltip text="namespace" term_id="namespace">}}. Every namespace gets a [`default` ServiceAccount](#default-service-accounts) upon creation. * **Lightweight:** Service accounts exist in the cluster and are defined in the Kubernetes API. You can quickly create service accounts to enable specific tasks. --> * **名字空间限定：** 每个服务账号都与一个 Kubernetes 名字空间绑定。 每个名字空间在创建时，会获得一个[名为 `default` 的 ServiceAccount](#default-service-accounts)。 * **轻量级：** 服务账号存在于集群中，并在 Kubernetes API 中定义。你可以快速创建服务账号以支持特定任务。 <!-- * **Portable:** A configuration bundle for a complex containerized workload might include service account definitions for the system's components. The lightweight nature of service accounts and the namespaced identities make the configurations portable. --> * **可移植性：** 复杂的容器化工作负载的配置包中可能包括针对系统组件的服务账号定义。 服务账号的轻量级性质和名字空间作用域的身份使得这类配置可移植。 <!-- Service accounts are different from user accounts, which are authenticated human users in the cluster. By default, user accounts don't exist in the Kubernetes API server; instead, the API server treats user identities as opaque data. You can authenticate as a user account using multiple methods. Some Kubernetes distributions might add custom extension APIs to represent user accounts in the API server. --> 服务账号与用户账号不同，用户账号是集群中通过了身份认证的人类用户。默认情况下， 用户账号不存在于 Kubernetes API 服务器中；相反，API 服务器将用户身份视为不透明数据。 你可以使用多种方法认证为某个用户账号。某些 Kubernetes 发行版可能会添加自定义扩展 API 来在 API 服务器中表示用户账号。 <!-- Comparison between service accounts and users --> {{< table caption="服务账号与用户之间的比较" >}} <!-- | Description | ServiceAccount | User or group | | --- | --- | --- | | Location | Kubernetes API (ServiceAccount object) | External | | Access control | Kubernetes RBAC or other [authorization mechanisms](/docs/reference/access-authn-authz/authorization/#authorization-modules) | Kubernetes RBAC or other identity and access management mechanisms | | Intended use | Workloads, automation | People | --> | 描述 | 服务账号 | 用户或组 | | --- | --- | --- | | 位置 | Kubernetes API（ServiceAccount 对象）| 外部 | | 访问控制 | Kubernetes RBAC 或其他[鉴权机制](/zh-cn/docs/reference/access-authn-authz/authorization/#authorization-modules) | Kubernetes RBAC 或其他身份和访问管理机制 | | 目标用途 | 工作负载、自动化工具 | 人 | {{< /table >}} <!-- ### Default service accounts {#default-service-accounts} --> ### 默认服务账号 {#default-service-accounts} <!-- When you create a cluster, Kubernetes automatically creates a ServiceAccount object named `default` for every namespace in your cluster. The `default` service accounts in each namespace get no permissions by default other than the [default API discovery permissions](/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings) that Kubernetes grants to all authenticated principals if role-based access control (RBAC) is enabled. If you delete the `default` ServiceAccount object in a namespace, the {{< glossary_tooltip text="control plane" term_id="control-plane" >}} replaces it with a new one. --> 在你创建集群时，Kubernetes 会自动为集群中的每个名字空间创建一个名为 `default` 的 ServiceAccount 对象。 在启用了基于角色的访问控制（RBAC）时，Kubernetes 为所有通过了身份认证的主体赋予 [默认 API 发现权限](/zh-cn/docs/reference/access-authn-authz/rbac/#default-roles-and-role-bindings)。 每个名字空间中的 `default` 服务账号除了这些权限之外，默认没有其他访问权限。 如果基于角色的访问控制（RBAC）被启用，当你删除名字空间中的 `default` ServiceAccount 对象时， {{< glossary_tooltip text="控制平面" term_id="control-plane" >}}会用新的 ServiceAccount 对象替换它。 <!-- If you deploy a Pod in a namespace, and you don't [manually assign a ServiceAccount to the Pod](#assign-to-pod), Kubernetes assigns the `default` ServiceAccount for that namespace to the Pod. --> 如果你在某个名字空间中部署 Pod，并且你没有[手动为 Pod 指派 ServiceAccount](#assign-to-pod)， Kubernetes 将该名字空间的 `default` 服务账号指派给这一 Pod。 <!-- ## Use cases for Kubernetes service accounts {#use-cases} As a general guideline, you can use service accounts to provide identities in the following scenarios: --> ## Kubernetes 服务账号的使用场景   {#use-cases} 一般而言，你可以在以下场景中使用服务账号来提供身份标识： <!-- * Your Pods need to communicate with the Kubernetes API server, for example in situations such as the following: * Providing read-only access to sensitive information stored in Secrets. * Granting [cross-namespace access](#cross-namespace), such as allowing a Pod in namespace `example` to read, list, and watch for Lease objects in the `kube-node-lease` namespace. --> * 你的 Pod 需要与 Kubernetes API 服务器通信，例如在以下场景中： * 提供对存储在 Secret 中的敏感信息的只读访问。 * 授予[跨名字空间访问](#cross-namespace)的权限，例如允许 `example` 名字空间中的 Pod 读取、列举和监视 `kube-node-lease` 名字空间中的 Lease 对象。 <!-- * Your Pods need to communicate with an external service. For example, a workload Pod requires an identity for a commercially available cloud API, and the commercial provider allows configuring a suitable trust relationship. * [Authenticating to a private image registry using an `imagePullSecret`](/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account). --> * 你的 Pod 需要与外部服务进行通信。例如，工作负载 Pod 需要一个身份来访问某商业化的云 API， 并且商业化 API 的提供商允许配置适当的信任关系。 * [使用 `imagePullSecret` 完成在私有镜像仓库上的身份认证](/zh-cn/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account)。 <!-- * An external service needs to communicate with the Kubernetes API server. For example, authenticating to the cluster as part of a CI/CD pipeline. * You use third-party security software in your cluster that relies on the ServiceAccount identity of different Pods to group those Pods into different contexts. --> * 外部服务需要与 Kubernetes API 服务器进行通信。例如，作为 CI/CD 流水线的一部分向集群作身份认证。 * 你在集群中使用了第三方安全软件，该软件依赖不同 Pod 的 ServiceAccount 身份，按不同上下文对这些 Pod 分组。 <!-- ## How to use service accounts {#how-to-use} To use a Kubernetes service account, you do the following: --> ## 如何使用服务账号  {#how-to-use} 要使用 Kubernetes 服务账号，你需要执行以下步骤： <!-- 1. Create a ServiceAccount object using a Kubernetes client like `kubectl` or a manifest that defines the object. 1. Grant permissions to the ServiceAccount object using an authorization mechanism such as [RBAC](/docs/reference/access-authn-authz/rbac/). --> 1. 使用像 `kubectl` 这样的 Kubernetes 客户端或定义对象的清单（manifest）创建 ServiceAccount 对象。 2. 使用鉴权机制（如 [RBAC](/zh-cn/docs/reference/access-authn-authz/rbac/)）为 ServiceAccount 对象授权。 <!-- 1. Assign the ServiceAccount object to Pods during Pod creation. If you're using the identity from an external service, [retrieve the ServiceAccount token](#get-a-token) and use it from that service instead. --> 3. 在创建 Pod 期间将 ServiceAccount 对象指派给 Pod。 如果你所使用的是来自外部服务的身份，可以[获取 ServiceAccount 令牌](#get-a-token)，并在该服务中使用这一令牌。 <!-- For instructions, refer to [Configure Service Accounts for Pods](/docs/tasks/configure-pod-container/configure-service-account/). --> 有关具体操作说明，参阅[为 Pod 配置服务账号](/zh-cn/docs/tasks/configure-pod-container/configure-service-account/)。 <!-- ### Grant permissions to a ServiceAccount {#grant-permissions} --> ### 为 ServiceAccount 授权   {#grant-permissions} <!-- You can use the built-in Kubernetes [role-based access control (RBAC)](/docs/reference/access-authn-authz/rbac/) mechanism to grant the minimum permissions required by each service account. You create a *role*, which grants access, and then *bind* the role to your ServiceAccount. RBAC lets you define a minimum set of permissions so that the service account permissions follow the principle of least privilege. Pods that use that service account don't get more permissions than are required to function correctly. --> 你可以使用 Kubernetes 内置的 [基于角色的访问控制 (RBAC)](/zh-cn/docs/reference/access-authn-authz/rbac/)机制来为每个服务账号授予所需的最低权限。 你可以创建一个用来授权的**角色**，然后将此角色**绑定**到你的 ServiceAccount 上。 RBAC 可以让你定义一组最低权限，使得服务账号权限遵循最小特权原则。 这样使用服务账号的 Pod 不会获得超出其正常运行所需的权限。 <!-- For instructions, refer to [ServiceAccount permissions](/docs/reference/access-authn-authz/rbac/#service-account-permissions). --> 有关具体操作说明，参阅 [ServiceAccount 权限](/zh-cn/docs/reference/access-authn-authz/rbac/#service-account-permissions)。 <!-- #### Cross-namespace access using a ServiceAccount {#cross-namespace} --> #### 使用 ServiceAccount 进行跨名字空间访问   {#cross-namespace} <!-- You can use RBAC to allow service accounts in one namespace to perform actions on resources in a different namespace in the cluster. For example, consider a scenario where you have a service account and Pod in the `dev` namespace and you want your Pod to see Jobs running in the `maintenance` namespace. You could create a Role object that grants permissions to list Job objects. Then, you'd create a RoleBinding object in the `maintenance` namespace to bind the Role to the ServiceAccount object. Now, Pods in the `dev` namespace can list Job objects in the `maintenance` namespace using that service account. --> 你可以使用 RBAC 允许一个名字空间中的服务账号对集群中另一个名字空间的资源执行操作。 例如，假设你在 `dev` 名字空间中有一个服务账号和一个 Pod，并且希望该 Pod 可以查看 `maintenance` 名字空间中正在运行的 Job。你可以创建一个 Role 对象来授予列举 Job 对象的权限。 随后在 `maintenance` 名字空间中创建 RoleBinding 对象将 Role 绑定到此 ServiceAccount 对象上。 现在，`dev` 名字空间中的 Pod 可以使用该服务账号列出 `maintenance` 名字空间中的 Job 对象集合。 <!-- ### Assign a ServiceAccount to a Pod {#assign-to-pod} To assign a ServiceAccount to a Pod, you set the `spec.serviceAccountName` field in the Pod specification. Kubernetes then automatically provides the credentials for that ServiceAccount to the Pod. In v1.22 and later, Kubernetes gets a short-lived, **automatically rotating** token using the `TokenRequest` API and mounts the token as a [projected volume](/docs/concepts/storage/projected-volumes/#serviceaccounttoken). --> ### 将 ServiceAccount 指派给 Pod   {#assign-to-pod} 要将某 ServiceAccount 指派给某 Pod，你需要在该 Pod 的规约中设置 `spec.serviceAccountName` 字段。 Kubernetes 将自动为 Pod 提供该 ServiceAccount 的凭据。在 Kubernetes v1.22 及更高版本中， Kubernetes 使用 `TokenRequest` API 获取一个短期的、**自动轮换**的令牌， 并以[投射卷](/zh-cn/docs/concepts/storage/projected-volumes/#serviceaccounttoken)的形式挂载此令牌。 <!-- By default, Kubernetes provides the Pod with the credentials for an assigned ServiceAccount, whether that is the `default` ServiceAccount or a custom ServiceAccount that you specify. To prevent Kubernetes from automatically injecting credentials for a specified ServiceAccount or the `default` ServiceAccount, set the `automountServiceAccountToken` field in your Pod specification to `false`. --> 默认情况下，Kubernetes 会将所指派的 ServiceAccount （无论是 `default` 服务账号还是你指定的定制 ServiceAccount）的凭据提供给 Pod。 要防止 Kubernetes 自动注入指定的 ServiceAccount 或 `default` ServiceAccount 的凭据， 可以将 Pod 规约中的 `automountServiceAccountToken` 字段设置为 `false`。 <!-- OK to remove this historical detail after Kubernetes 1.31 is released --> <!-- In versions earlier than 1.22, Kubernetes provides a long-lived, static token to the Pod as a Secret. --> 在 Kubernetes 1.22 之前的版本中，Kubernetes 会将一个长期有效的静态令牌以 Secret 形式提供给 Pod。 <!-- #### Manually retrieve ServiceAccount credentials {#get-a-token} If you need the credentials for a ServiceAccount to mount in a non-standard location, or for an audience that isn't the API server, use one of the following methods: --> #### 手动获取 ServiceAccount 凭据   {#get-a-token} 如果你需要 ServiceAccount 的凭据并将其挂载到非标准位置，或者用于 API 服务器之外的受众，可以使用以下方法之一： <!-- * [TokenRequest API](/docs/reference/kubernetes-api/authentication-resources/token-request-v1/) (recommended): Request a short-lived service account token from within your own *application code*. The token expires automatically and can rotate upon expiration. If you have a legacy application that is not aware of Kubernetes, you could use a sidecar container within the same pod to fetch these tokens and make them available to the application workload. --> * [TokenRequest API](/zh-cn/docs/reference/kubernetes-api/authentication-resources/token-request-v1/)（推荐）： 在你自己的**应用代码**中请求一个短期的服务账号令牌。此令牌会自动过期，并可在过期时被轮换。 如果你有一个旧的、对 Kubernetes 无感知能力的应用，你可以在同一个 Pod 内使用边车容器来获取这些令牌，并将其提供给应用工作负载。 <!-- * [Token Volume Projection](/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection) (also recommended): In Kubernetes v1.20 and later, use the Pod specification to tell the kubelet to add the service account token to the Pod as a *projected volume*. Projected tokens expire automatically, and the kubelet rotates the token before it expires. --> * [令牌卷投射](/zh-cn/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection)（同样推荐）： 在 Kubernetes v1.20 及更高版本中，使用 Pod 规约告知 kubelet 将服务账号令牌作为**投射卷**添加到 Pod 中。 所投射的令牌会自动过期，在过期之前 kubelet 会自动轮换此令牌。 <!-- * [Service Account Token Secrets](/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount) (not recommended): You can mount service account tokens as Kubernetes Secrets in Pods. These tokens don't expire and don't rotate. This method is not recommended, especially at scale, because of the risks associated with static, long-lived credentials. In Kubernetes v1.24 and later, the [LegacyServiceAccountTokenNoAutoGeneration feature gate](/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-graduated-or-deprecated-features) prevents Kubernetes from automatically creating these tokens for ServiceAccounts. `LegacyServiceAccountTokenNoAutoGeneration` is enabled by default; in other words, Kubernetes does not create these tokens. --> * [服务账号令牌 Secret](/zh-cn/docs/tasks/configure-pod-container/configure-service-account/#manually-create-an-api-token-for-a-serviceaccount)（不推荐）： 你可以将服务账号令牌以 Kubernetes Secret 的形式挂载到 Pod 中。这些令牌不会过期且不会轮换。 不推荐使用此方法，特别是在大规模场景下，这是因为静态、长期有效的凭据存在一定的风险。在 Kubernetes v1.24 及更高版本中， [LegacyServiceAccountTokenNoAutoGeneration 特性门控](/zh-cn/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-graduated-or-deprecated-features)阻止 Kubernetes 自动为 ServiceAccount 创建这些令牌。`LegacyServiceAccountTokenNoAutoGeneration` 默认被启用， 也就是说，Kubernetes 不会创建这些令牌。 {{< note >}} <!-- For applications running outside your Kubernetes cluster, you might be considering creating a long-lived ServiceAccount token that is stored in a Secret. This allows authentication, but the Kubernetes project recommends you avoid this approach. Long-lived bearer tokens represent a security risk as, once disclosed, the token can be misused. Instead, consider using an alternative. For example, your external application can authenticate using a well-protected private key `and` a certificate, or using a custom mechanism such as an [authentication webhook](/docs/reference/access-authn-authz/authentication/#webhook-token-authentication) that you implement yourself. --> 对于运行在 Kubernetes 集群外的应用，你可能考虑创建一个长期有效的 ServiceAccount 令牌， 并将其存储在 Secret 中。尽管这种方式可以实现身份认证，但 Kubernetes 项目建议你避免使用此方法。 长期有效的持有者令牌（Bearer Token）会带来安全风险，一旦泄露，此令牌就可能被滥用。 为此，你可以考虑使用其他替代方案。例如，你的外部应用可以使用一个保护得很好的私钥和证书进行身份认证， 或者使用你自己实现的[身份认证 Webhook](/zh-cn/docs/reference/access-authn-authz/authentication/#webhook-token-authentication) 这类自定义机制。 <!-- You can also use TokenRequest to obtain short-lived tokens for your external application. --> 你还可以使用 TokenRequest 为外部应用获取短期的令牌。 {{< /note >}} <!-- ## Authenticating service account credentials {#authenticating-credentials} --> ## 对服务账号凭据进行鉴别   {#authenticating-credentials} <!-- ServiceAccounts use signed {{<glossary_tooltip term_id="jwt" text="JSON Web Tokens">}}  (JWTs) to authenticate to the Kubernetes API server, and to any other system where a trust relationship exists. Depending on how the token was issued (either time-limited using a `TokenRequest` or using a legacy mechanism with a Secret), a ServiceAccount token might also have an expiry time, an audience, and a time after which the token *starts* being valid. When a client that is acting as a ServiceAccount tries to communicate with the Kubernetes API server, the client includes an `Authorization: Bearer <token>` header with the HTTP request. The API server checks the validity of that bearer token as follows: --> ServiceAccount 使用签名的 JSON Web Token (JWT) 来向 Kubernetes API 服务器以及任何其他存在信任关系的系统进行身份认证。根据令牌的签发方式 （使用 `TokenRequest` 限制时间或使用传统的 Secret 机制），ServiceAccount 令牌也可能有到期时间、受众和令牌**开始**生效的时间点。 当客户端以 ServiceAccount 的身份尝试与 Kubernetes API 服务器通信时， 客户端会在 HTTP 请求中包含 `Authorization: Bearer <token>` 标头。 API 服务器按照以下方式检查该持有者令牌的有效性： <!-- 1. Checks the token signature. 1. Checks whether the token has expired. 1. Checks whether object references in the token claims are currently valid. 1. Checks whether the token is currently valid. 1. Checks the audience claims. --> 1. 检查令牌签名。 1. 检查令牌是否已过期。 1. 检查令牌申明中的对象引用是否当前有效。 1. 检查令牌是否当前有效。 1. 检查受众申明。 <!-- The TokenRequest API produces _bound tokens_ for a ServiceAccount. This binding is linked to the lifetime of the client, such as a Pod, that is acting as that ServiceAccount. For tokens issued using the `TokenRequest` API, the API server also checks that the specific object reference that is using the ServiceAccount still exists, matching by the {{< glossary_tooltip term_id="uid" text="unique ID" >}} of that object. For legacy tokens that are mounted as Secrets in Pods, the API server checks the token against the Secret. --> TokenRequest API 为 ServiceAccount 生成**绑定令牌**。这种绑定与以该 ServiceAccount 身份运行的 的客户端（如 Pod）的生命期相关联。 对于使用 `TokenRequest` API 签发的令牌，API 服务器还会检查正在使用 ServiceAccount 的特定对象引用是否仍然存在， 方式是通过该对象的{{< glossary_tooltip term_id="uid" text="唯一 ID" >}} 进行匹配。 对于以 Secret 形式挂载到 Pod 中的旧有令牌，API 服务器会基于 Secret 来检查令牌。 <!-- For more information about the authentication process, refer to [Authentication](/docs/reference/access-authn-authz/authentication/#service-account-tokens). --> 有关身份认证过程的更多信息，参考[身份认证](/zh-cn/docs/reference/access-authn-authz/authentication/#service-account-tokens)。 <!-- ### Authenticating service account credentials in your own code {#authenticating-in-code} If you have services of your own that need to validate Kubernetes service account credentials, you can use the following methods: --> ### 在自己的代码中检查服务账号凭据   {#authenticating-in-code} 如果你的服务需要检查 Kubernetes 服务账号凭据，可以使用以下方法： <!-- * [TokenReview API](/docs/reference/kubernetes-api/authentication-resources/token-review-v1/) (recommended) * OIDC discovery --> * [TokenReview API](/zh-cn/docs/reference/kubernetes-api/authentication-resources/token-review-v1/)（推荐） * OIDC 发现 <!-- The Kubernetes project recommends that you use the TokenReview API, because this method invalidates tokens that are bound to API objects such as Secrets, ServiceAccounts, and Pods when those objects are deleted. For example, if you delete the Pod that contains a projected ServiceAccount token, the cluster invalidates that token immediately and a TokenReview immediately fails. If you use OIDC validation instead, your clients continue to treat the token as valid until the token reaches its expiration timestamp. --> Kubernetes 项目建议你使用 TokenReview API，因为当你删除某些 API 对象 （如 Secret、ServiceAccount 和 Pod）的时候，此方法将使绑定到这些 API 对象上的令牌失效。 例如，如果删除包含投射 ServiceAccount 令牌的 Pod，则集群立即使该令牌失效， 并且 TokenReview 操作也会立即失败。 如果你使用的是 OIDC 验证，则客户端将继续将令牌视为有效，直到令牌达到其到期时间戳。 <!-- Your application should always define the audience that it accepts, and should check that the token's audiences match the audiences that the application expects. This helps to minimize the scope of the token so that it can only be used in your application and nowhere else. --> 你的应用应始终定义其所接受的受众，并检查令牌的受众是否与应用期望的受众匹配。 这有助于将令牌的作用域最小化，这样它只能在你的应用内部使用，而不能在其他地方使用。 <!-- ## Alternatives * Issue your own tokens using another mechanism, and then use [Webhook Token Authentication](/docs/reference/access-authn-authz/authentication/#webhook-token-authentication) to validate bearer tokens using your own validation service. --> ## 替代方案   {#alternatives} * 使用其他机制签发你自己的令牌，然后使用 [Webhook 令牌身份认证](/zh-cn/docs/reference/access-authn-authz/authentication/#webhook-token-authentication)通过你自己的验证服务来验证持有者令牌。 <!-- * Provide your own identities to Pods. * [Use the SPIFFE CSI driver plugin to provide SPIFFE SVIDs as X.509 certificate pairs to Pods](https://cert-manager.io/docs/projects/csi-driver-spiffe/). {{% thirdparty-content single="true" %}} * [Use a service mesh such as Istio to provide certificates to Pods](https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/). --> * 为 Pod 提供你自己的身份： * [使用 SPIFFE CSI 驱动插件将 SPIFFE SVID 作为 X.509 证书对提供给 Pod](https://cert-manager.io/docs/projects/csi-driver-spiffe/)。 {{% thirdparty-content single="true" %}} * [使用 Istio 这类服务网格为 Pod 提供证书](https://istio.io/latest/zh/docs/tasks/security/cert-management/plugin-ca-cert/)。 <!-- * Authenticate from outside the cluster to the API server without using service account tokens: * [Configure the API server to accept OpenID Connect (OIDC) tokens from your identity provider](/docs/reference/access-authn-authz/authentication/#openid-connect-tokens). * Use service accounts or user accounts created using an external Identity and Access Management (IAM) service, such as from a cloud provider, to authenticate to your cluster. * [Use the CertificateSigningRequest API with client certificates](/docs/tasks/tls/managing-tls-in-a-cluster/). --> * 从集群外部向 API 服务器进行身份认证，而不使用服务账号令牌： * [配置 API 服务器接受来自你自己的身份驱动的 OpenID Connect (OIDC) 令牌](/zh-cn/docs/reference/access-authn-authz/authentication/#openid-connect-tokens)。 * 使用来自云提供商等外部身份和访问管理 (IAM) 服务创建的服务账号或用户账号向集群进行身份认证。 * [使用 CertificateSigningRequest API 和客户端证书](/zh-cn/docs/tasks/tls/managing-tls-in-a-cluster/)。 <!-- * [Configure the kubelet to retrieve credentials from an image registry](/docs/tasks/administer-cluster/kubelet-credential-provider/). * Use a Device Plugin to access a virtual Trusted Platform Module (TPM), which then allows authentication using a private key. --> * [配置 kubelet 从镜像仓库中获取凭据](/zh-cn/docs/tasks/administer-cluster/kubelet-credential-provider/)。 * 使用设备插件访问虚拟的可信平台模块 (TPM)，进而可以使用私钥进行身份认证。 ## {{% heading "whatsnext" %}} <!-- * Learn how to [manage your ServiceAccounts as a cluster administrator](/docs/reference/access-authn-authz/service-accounts-admin/). * Learn how to [assign a ServiceAccount to a Pod](/docs/tasks/configure-pod-container/configure-service-account/). * Read the [ServiceAccount API reference](/docs/reference/kubernetes-api/authentication-resources/service-account-v1/). --> * 学习如何[作为集群管理员管理你的 ServiceAccount](/zh-cn/docs/reference/access-authn-authz/service-accounts-admin/)。 * 学习如何[将 ServiceAccount 指派给 Pod](/zh-cn/docs/tasks/configure-pod-container/configure-service-account/)。 * 阅读 [ServiceAccount API 参考文档](/zh-cn/docs/reference/kubernetes-api/authentication-resources/service-account-v1/)。 </ins>

The file is not translated in the zh-cn localization.
/sig docs /language zh /area localization
/triage accepted
/assign


Question: <ins> --- title: Multi-tenancy content_type: concept weight: 70 --- <!-- overview --> This page provides an overview of available configuration options and best practices for cluster multi-tenancy. Sharing clusters saves costs and simplifies administration. However, sharing clusters also presents challenges such as security, fairness, and managing _noisy neighbors_. Clusters can be shared in many ways. In some cases, different applications may run in the same cluster. In other cases, multiple instances of the same application may run in the same cluster, one for each end user. All these types of sharing are frequently described using the umbrella term _multi-tenancy_. While Kubernetes does not have first-class concepts of end users or tenants, it provides several features to help manage different tenancy requirements. These are discussed below. <!-- body --> ## Use cases The first step to determining how to share your cluster is understanding your use case, so you can evaluate the patterns and tools available. In general, multi-tenancy in Kubernetes clusters falls into two broad categories, though many variations and hybrids are also possible. ### Multiple teams A common form of multi-tenancy is to share a cluster between multiple teams within an organization, each of whom may operate one or more workloads. These workloads frequently need to communicate with each other, and with other workloads located on the same or different clusters. In this scenario, members of the teams often have direct access to Kubernetes resources via tools such as `kubectl`, or indirect access through GitOps controllers or other types of release automation tools. There is often some level of trust between members of different teams, but Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly share clusters. ### Multiple customers The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor running multiple instances of a workload for customers. This business model is so strongly associated with this deployment style that many people call it "SaaS tenancy." However, a better term might be "multi-customer tenancy,” since SaaS vendors may also use other deployment models, and this deployment model can also be used outside of SaaS. In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from their perspective and is only used by the vendor to manage the workloads. Cost optimization is frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are strongly isolated from each other. ## Terminology ### Tenants When discussing multi-tenancy in Kubernetes, there is no single definition for a "tenant". Rather, the definition of a tenant will vary depending on whether multi-team or multi-customer tenancy is being discussed. In multi-team usage, a tenant is typically a team, where each team typically deploys a small number of workloads that scales with the complexity of the service. However, the definition of "team" may itself be fuzzy, as teams may be organized into higher-level divisions or subdivided into smaller teams. By contrast, if each team deploys dedicated workloads for each new client, they are using a multi-customer model of tenancy. In this case, a "tenant" is simply a group of users who share a single workload. This may be as large as an entire company, or as small as a single team at that company. In many cases, the same organization may use both definitions of "tenants" in different contexts. For example, a platform team may offer shared services such as security tools and databases to multiple internal “customers” and a SaaS vendor may also have multiple teams sharing a development cluster. Finally, hybrid architectures are also possible, such as a SaaS provider using a combination of per-customer workloads for sensitive data, combined with multi-tenant shared services. {{< figure src="/images/docs/multi-tenancy.png" title="A cluster showing coexisting tenancy models" class="diagram-large" >}} ### Isolation There are several ways to design and build multi-tenant solutions with Kubernetes. Each of these methods comes with its own set of tradeoffs that impact the isolation level, implementation effort, operational complexity, and cost of service. A Kubernetes cluster consists of a control plane which runs Kubernetes software, and a data plane consisting of worker nodes where tenant workloads are executed as pods. Tenant isolation can be applied in both the control plane and the data plane based on organizational requirements. The level of isolation offered is sometimes described using terms like “hard” multi-tenancy, which implies strong isolation, and “soft” multi-tenancy, which implies weaker isolation. In particular, "hard" multi-tenancy is often used to describe cases where the tenants do not trust each other, often from security and resource sharing perspectives (e.g. guarding against attacks such as data exfiltration or DoS). Since data planes typically have much larger attack surfaces, "hard" multi-tenancy often requires extra attention to isolating the data-plane, though control plane isolation  also remains critical. However, the terms "hard" and "soft" can often be confusing, as there is no single definition that will apply to all users. Rather, "hardness" or "softness" is better understood as a broad spectrum, with many different techniques that can be used to maintain different types of isolation in your clusters, based on your requirements. In more extreme cases, it may be easier or necessary to forgo any cluster-level sharing at all and assign each tenant their dedicated cluster, possibly even running on dedicated hardware if VMs are not considered an adequate security boundary. This may be easier with managed Kubernetes clusters, where the overhead of creating and operating clusters is at least somewhat taken on by a cloud provider. The benefit of stronger tenant isolation must be evaluated against the cost and complexity of managing multiple clusters. The [Multi-cluster SIG](https://git.k8s.io/community/sig-multicluster/README.md) is responsible for addressing these types of use cases. The remainder of this page focuses on isolation techniques used for shared Kubernetes clusters. However, even if you are considering dedicated clusters, it may be valuable to review these recommendations, as it will give you the flexibility to shift to shared clusters in the future if your needs or capabilities change. ## Control plane isolation Control plane isolation ensures that different tenants cannot access or affect each others' Kubernetes API resources. ### Namespaces In Kubernetes, a {{< glossary_tooltip text="Namespace" term_id="namespace" >}} provides a mechanism for isolating groups of API resources within a single cluster. This isolation has two key dimensions: 1. Object names within a namespace can overlap with names in other namespaces, similar to files in folders. This allows tenants to name their resources without having to consider what other tenants are doing. 2. Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be restricted to a namespace. In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and distinct management unit. In fact, a common practice is to isolate every workload in its own namespace, even if multiple workloads are operated by the same tenant. This ensures that each workload has its own identity and can be configured with an appropriate security policy. The namespace isolation model requires configuration of several other Kubernetes resources, networking plugins, and adherence to security best practices to properly isolate tenant workloads. These considerations are discussed below. ### Access controls The most important type of isolation for the control plane is authorization. If teams or their workloads can access or modify each others' API resources, they can change or disable all other types of policies thereby negating any protection those policies may offer. As a result, it is critical to ensure that each tenant has the appropriate access to only the namespaces they need, and no more. This is known as the "Principle of Least Privilege." Role-based access control (RBAC) is commonly used to enforce authorization in the Kubernetes control plane, for both users and workloads (service accounts). [Roles](/docs/reference/access-authn-authz/rbac/#role-and-clusterrole) and [role bindings](/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) are Kubernetes objects that are used at a namespace level to enforce access control in your application; similar objects exist for authorizing access to cluster-level objects, though these are less useful for multi-tenant clusters. In a multi-team environment, RBAC must be used to restrict tenants' access to the appropriate namespaces, and ensure that cluster-wide resources can only be accessed or modified by privileged users such as cluster administrators. If a policy ends up granting a user more permissions than they need, this is likely a signal that the namespace containing the affected resources should be refactored into finer-grained namespaces. Namespace management tools may simplify the management of these finer-grained namespaces by applying common RBAC policies to different namespaces, while still allowing fine-grained policies where necessary. ### Quotas Kubernetes workloads consume node resources, like CPU and memory.  In a multi-tenant environment, you can use [Resource Quotas](/docs/concepts/policy/resource-quotas/) to manage resource usage of tenant workloads. For the multiple teams use case, where tenants have access to the Kubernetes API, you can use resource quotas to limit the number of API resources (for example: the number of Pods, or the number of ConfigMaps) that a tenant can create. Limits on object count ensure fairness and aim to avoid _noisy neighbor_ issues from affecting other tenants that share a control plane. Resource quotas are namespaced objects. By mapping tenants to namespaces, cluster admins can use quotas to ensure that a tenant cannot monopolize a cluster's resources or overwhelm its control plane. Namespace management tools simplify the administration of quotas. In addition, while Kubernetes quotas only apply within a single namespace, some namespace management tools allow groups of namespaces to share quotas, giving administrators far more flexibility with less effort than built-in quotas. Quotas prevent a single tenant from consuming greater than their allocated share of resources hence minimizing the “noisy neighbor” issue, where one tenant negatively impacts the performance of other tenants' workloads. When you apply a quota to namespace, Kubernetes requires you to also specify resource requests and limits for each container. Limits are the upper bound for the amount of resources that a container can consume. Containers that attempt to consume resources that exceed the configured limits will either be throttled or killed, based on the resource type. When resource requests are set lower than limits, each container is guaranteed the requested amount but there may still be some potential for impact across workloads. Quotas cannot protect against all kinds of resource sharing, such as network traffic. Node isolation (described below) may be a better solution for this problem. ## Data Plane Isolation Data plane isolation ensures that pods and workloads for different tenants are sufficiently isolated. ### Network isolation By default, all pods in a Kubernetes cluster are allowed to communicate with each other, and all network traffic is unencrypted. This can lead to security vulnerabilities where traffic is accidentally or maliciously sent to an unintended destination, or is intercepted by a workload on a compromised node. Pod-to-pod communication can be controlled  using [Network Policies](/docs/concepts/services-networking/network-policies/), which restrict communication between pods using namespace labels or IP address ranges. In a multi-tenant environment where strict network isolation between tenants is required,   starting with a default policy that denies communication between pods is recommended with another rule that allows all pods to query the DNS server for name resolution. With such a default policy in place, you can begin adding more permissive rules that allow for communication within a namespace. This scheme can be further refined as required. Note that this only applies to pods within a single control plane; pods that belong to different virtual control planes cannot talk to each other via Kubernetes networking. Namespace management tools may simplify the creation of default or common network policies. In addition, some of these tools allow you to enforce a consistent set of namespace labels across your cluster, ensuring that they are a trusted basis for your policies. {{< warning >}} Network policies require a [CNI plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#cni) that supports the implementation of network policies. Otherwise, NetworkPolicy resources will be ignored. {{< /warning >}} More advanced network isolation may be provided by service meshes, which provide OSI Layer 7 policies based on workload identity, in addition to namespaces. These higher-level policies can make it easier to manage namespaced based multi-tenancy, especially when multiple namespaces are dedicated to a single tenant. They frequently also offer encryption using mutual TLS, protecting your data even in the presence of a compromised node, and work across dedicated or virtual clusters. However, they can be significantly more complex to manage and may not be appropriate for all users. ### Storage isolation Kubernetes offers several types of volumes that can be used as persistent storage for workloads. For security and data-isolation, [dynamic volume provisioning](/docs/concepts/storage/dynamic-provisioning/) is recommended and volume types that use node resources should be avoided. [StorageClasses](/docs/concepts/storage/storage-classes/) allow you to describe custom "classes" of storage offered by your cluster, based on quality-of-service levels, backup policies, or custom policies determined by the cluster administrators. Pods can request storage using a [PersistentVolumeClaim](/docs/concepts/storage/persistent-volumes/). A PersistentVolumeClaim is a namespaced resource, which enables isolating portions of the storage system and dedicating it to tenants within the shared Kubernetes cluster. However, it is important to note that a PersistentVolume is a cluster-wide resource and has a lifecycle independent of workloads and namespaces. For example, you can configure a separate StorageClass for each tenant and use this to strengthen isolation. If a StorageClass is shared, you should set a [reclaim policy of `Delete`](/docs/concepts/storage/storage-classes/#reclaim-policy) to ensure that a PersistentVolume cannot be reused across different namespaces. ### Sandboxing containers {{% thirdparty-content %}} Kubernetes pods are composed of one or more containers that execute on worker nodes. Containers utilize OS-level virtualization and hence offer a weaker isolation boundary than virtual machines that utilize hardware-based virtualization. In a shared environment, unpatched vulnerabilities in the application and system layers can be exploited by attackers for container breakouts and remote code execution that allow access to host resources. In some applications, like a Content Management System (CMS), customers may be allowed the ability to upload and execute untrusted scripts or code. In either case, mechanisms to further isolate and protect workloads using strong isolation are desirable. Sandboxing provides a way to isolate workloads running in a shared cluster. It typically involves running each pod in a separate execution environment such as a virtual machine or a userspace kernel. Sandboxing is often recommended when you are running untrusted code, where workloads are assumed to be malicious. Part of the reason this type of isolation is necessary is because containers are processes running on a shared kernel; they mount file systems like /sys and /proc from the underlying host, making them less secure than an application that runs on a virtual machine which has its own kernel. While controls such as seccomp, AppArmor, and SELinux can be used to strengthen the security of containers, it is hard to apply a universal set of rules to all workloads running in a shared cluster. Running workloads in a sandbox environment helps to insulate the host from container escapes, where an attacker exploits a vulnerability to gain access to the host system and all the processes/files running on that host. Virtual machines and userspace kernels are 2 popular approaches to sandboxing. The following sandboxing implementations are available: * [gVisor](https://gvisor.dev/) intercepts syscalls from containers and runs them through a userspace kernel, written in Go, with limited access to the underlying host. * [Kata Containers](https://katacontainers.io/) is an OCI compliant runtime that allows you to run containers in a VM. The hardware virtualization available in Kata offers an added layer of security for containers running untrusted code. ### Node Isolation Node isolation is another technique that you can use to isolate tenant workloads from each other. With node isolation, a set of nodes is dedicated to running pods from a particular tenant and co-mingling of tenant pods is prohibited. This configuration reduces the noisy tenant issue, as all pods running on a node will belong to a single tenant. The risk of information disclosure is slightly lower with node isolation because an attacker that manages to escape from a container will only have access to the containers and volumes mounted to that node. Although workloads from different tenants are running on different nodes, it is important to be aware that the kubelet and (unless using virtual control planes) the API service are still shared services. A skilled attacker could use the permissions assigned to the kubelet or other pods running on the node to move laterally within the cluster and gain access to tenant workloads running on other nodes. If this is a major concern, consider implementing compensating controls such as seccomp, AppArmor or SELinux or explore using sandboxed containers or creating separate clusters for each tenant. Node isolation is a little easier to reason about from a billing standpoint than sandboxing containers since you can charge back per node rather than per pod. It also has fewer compatibility and performance issues and may be easier to implement than sandboxing containers. For example, nodes for each tenant can be configured with taints so that only pods with the corresponding toleration can run on them. A mutating webhook could then be used to automatically add tolerations and node affinities to pods deployed into tenant namespaces so that they run on a specific set of nodes designated for that tenant. Node isolation can be implemented using an [pod node selectors](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) or a [Virtual Kubelet](https://github.com/virtual-kubelet). ## Additional Considerations This section discusses other Kubernetes constructs and patterns that are relevant for multi-tenancy. ### API Priority and Fairness [API priority and fairness](/docs/concepts/cluster-administration/flow-control/) is a Kubernetes feature that allows you to assign a priority to certain pods running within the cluster. When an application calls the Kubernetes API, the API server evaluates the priority assigned to pod. Calls from pods with higher priority are fulfilled before those with a lower priority. When contention is high, lower priority calls can be queued until the server is less busy or you can reject the requests. Using API priority and fairness will not be very common in SaaS environments unless you are allowing customers to run applications that interface with the Kubernetes API, e.g. a controller. ### Quality-of-Service (QoS) {#qos} When you’re running a SaaS application, you may want the ability to offer different Quality-of-Service (QoS) tiers of service to different tenants. For example, you may have freemium service that comes with fewer performance guarantees and features and a for-fee service tier with specific performance guarantees. Fortunately, there are several Kubernetes constructs that can help you accomplish this within a shared cluster, including network QoS, storage classes, and pod priority and preemption. The idea with each of these is to provide tenants with the quality of service that they paid for. Let’s start by looking at networking QoS. Typically, all pods on a node share a network interface. Without network QoS, some pods may consume an unfair share of the available bandwidth at the expense of other pods. The Kubernetes [bandwidth plugin](https://www.cni.dev/plugins/current/meta/bandwidth/) creates an [extended resource](/docs/concepts/configuration/manage-resources-containers/#extended-resources) for networking that allows you to use Kubernetes resources constructs, i.e. requests/limits, to apply rate limits to pods by using Linux tc queues. Be aware that the plugin is considered experimental as per the [Network Plugins](/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/#support-traffic-shaping) documentation and should be thoroughly tested before use in production environments. For storage QoS, you will likely want to create different storage classes or profiles with different performance characteristics. Each storage profile can be associated with a different tier of service that is optimized for different workloads such IO, redundancy, or throughput. Additional logic might be necessary to allow the tenant to associate the appropriate storage profile with their workload. Finally, there’s [pod priority and preemption](/docs/concepts/scheduling-eviction/pod-priority-preemption/) where you can assign priority values to pods. When scheduling pods, the scheduler will try evicting pods with lower priority when there are insufficient resources to schedule pods that are assigned a higher priority. If you have a use case where tenants have different service tiers in in a shared cluster e.g. free and paid, you may want to give higher priority to certain tiers using this feature. ### DNS Kubernetes clusters include a Domain Name System (DNS) service to provide translations from names to IP addresses, for all Services and Pods. By default, the Kubernetes DNS service allows lookups across all namespaces in the cluster. In multi-tenant environments where tenants can access pods and other Kubernetes resources, or where stronger isolation is required, it may be necessary to prevent pods from looking up services in other Namespaces. You can restrict cross-namespace DNS lookups by configuring security rules for the DNS service. For example, CoreDNS (the default DNS service for Kubernetes) can leverage Kubernetes metadata to restrict queries to Pods and Services within a namespace. For more information, read an [example](https://github.com/coredns/policy#kubernetes-metadata-multi-tenancy-policy) of configuring this within the CoreDNS documentation. When a [Virtual Control Plane per tenant](#virtual-control-plane-per-tenant) model is used, a DNS service must be configured per tenant or a multi-tenant DNS service must be used. Here is an example of a [customized version of CoreDNS](https://github.com/kubernetes-sigs/cluster-api-provider-nested/blob/main/virtualcluster/doc/tenant-dns.md) that supports multiple tenants. ### Operators [Operators](/docs/concepts/extend-kubernetes/operator/) are Kubernetes controllers that manage applications. Operators can simplify the management of multiple instances of an application, like a database service, which makes them a common building block in the multi-consumer (SaaS) multi-tenancy use case. Operators used in a multi-tenant environment should follow a stricter set of guidelines. Specifically, the Operator should: * Support creating resources within different tenant namespaces, rather than just in the namespace in which the Operator is deployed. * Ensure that the Pods are configured with resource requests and limits, to ensure scheduling and fairness. * Support configuration of Pods for data-plane isolation techniques such as node isolation and sandboxed containers. ## Implementations {{% thirdparty-content %}} There are two primary ways to share a Kubernetes cluster for multi-tenancy: using Namespaces (i.e. a Namespace per tenant) or by virtualizing the control plane (i.e. Virtual control plane per tenant). In both cases, data plane isolation, and management of additional considerations such as API Priority and Fairness, is also recommended. Namespace isolation is well-supported by Kubernetes, has a negligible resource cost, and provides mechanisms to allow tenants to interact appropriately, such as by allowing service-to-service communication. However, it can be difficult to configure, and doesn't apply to Kubernetes resources that can't be namespaced, such as Custom Resource Definitions, Storage Classes, and Webhooks. Control plane virtualization allows for isolation of non-namespaced resources at the cost of somewhat higher resource usage and more difficult cross-tenant sharing. It is a good option when namespace isolation is insufficient but dedicated clusters are undesirable, due to the high cost of maintaining them (especially on-prem) or due to their higher overhead and lack of resource sharing. However, even within a virtualized control plane, you will likely see benefits by using namespaces as well. The two options are discussed in more detail in the following sections: ### Namespace per tenant As previously mentioned, you should consider isolating each workload in its own namespace, even if you are using dedicated clusters or virtualized control planes. This ensures that each workload only has access to its own resources, such as Config Maps and Secrets, and allows you to tailor dedicated security policies for each workload. In addition, it is a best practice to give each namespace names that are unique across your entire fleet (i.e., even if they are in separate clusters), as this gives you the flexibility to switch between dedicated and shared clusters in the future, or to use multi-cluster tooling such as service meshes. Conversely, there are also advantages to assigning namespaces at the tenant level, not just the workload level, since there are often policies that apply to all workloads owned by a single tenant. However, this raises its own problems. Firstly, this makes it difficult or impossible to customize policies to individual workloads, and secondly, it may be challenging to come up with a single level of "tenancy" that should be given a namespace. For example, an organization may have divisions, teams, and subteams - which should be assigned a namespace? To solve this, Kubernetes provides the [Hierarchical Namespace Controller (HNC)](https://github.com/kubernetes-sigs/hierarchical-namespaces), which allows you to organize your namespaces into hierarchies, and share certain policies and resources between them. It also helps you manage namespace labels, namespace lifecycles, and delegated management, and share resource quotas across related namespaces. These capabilities can be useful in both multi-team and multi-customer scenarios. Other projects that provide similar capabilities and aid in managing namespaced resources are listed below: #### Multi-team tenancy * [Capsule](https://github.com/clastix/capsule) * [Kiosk](https://github.com/loft-sh/kiosk) #### Multi-customer tenancy * [Kubeplus](https://github.com/cloud-ark/kubeplus) #### Policy engines Policy engines provide features to validate and generate tenant configurations: * [Kyverno](https://kyverno.io/) * [OPA/Gatekeeper](https://github.com/open-policy-agent/gatekeeper) ### Virtual control plane per tenant Another form of control-plane isolation is to use Kubernetes extensions to provide each tenant a virtual control-plane that enables segmentation of cluster-wide API resources. [Data plane isolation](#data-plane-isolation) techniques can be used with this model to securely manage worker nodes across tenants. The virtual control plane based multi-tenancy model extends namespace-based multi-tenancy by providing each tenant with dedicated control plane components, and hence complete control over cluster-wide resources and add-on services. Worker nodes are shared across all tenants, and are managed by a Kubernetes cluster that is normally inaccessible to tenants. This cluster is often referred to as a _super-cluster_ (or sometimes as a _host-cluster_). Since a tenant’s control-plane is not directly associated with underlying compute resources it is referred to as a _virtual control plane_. A virtual control plane typically consists of the Kubernetes API server, the controller manager, and the etcd data store. It interacts with the super cluster via a metadata synchronization controller which coordinates changes across tenant control planes and the control plane of the super--cluster. By using per-tenant dedicated control planes, most of the isolation problems due to sharing one API server among all tenants are solved. Examples include noisy neighbors in the control plane, uncontrollable blast radius of policy misconfigurations, and conflicts between cluster scope objects such as webhooks and CRDs.  Hence, the virtual control plane model is particularly suitable for cases where each tenant requires access to a Kubernetes API server and expects the full cluster manageability. The improved isolation comes at the  cost of running and maintaining an individual virtual control plane per tenant. In addition, per-tenant control planes do not solve isolation problems in the data plane, such as node-level noisy neighbors or security threats. These must still be addressed separately. The Kubernetes [Cluster API - Nested (CAPN)](https://github.com/kubernetes-sigs/cluster-api-provider-nested/tree/main/virtualcluster) project provides an implementation of virtual control planes. #### Other implementations * [Kamaji](https://github.com/clastix/kamaji) * [vcluster](https://github.com/loft-sh/vcluster) </ins>

This is a Feature Request What would you like to be A section in the Kubernetes documentation to provide users with best practice guidelines and references to Kubernetes projects and solutions for multi-tenancy. This can be a new section in . Why is this needed Kubernetes is commonly used by multiple teams and to run multiple applications. The has been chartered with defining multi-tenancy models for Kubernetes and making these easy to consume for end users. The group has produced multiple projects like , , and and published blog posts on and the projects (, ). The HNC and Virtual Cluster projects have “graduated” from the working group and are now managed under SIG-auth and SIG-cluster-lifecycle. However, multi-tenancy remains a confusing and complex topic for Kubernetes users, especially beginners. Having a section on multi-tenancy in the Kubernetes documentation, with links to projects, would be helpful to all users. Comments If this seems like a good idea, the Multi-Tenancy WG can take the lead to propose a draft document. cc:
Happy for multitenancy WG to take this forward - also, that WG is welcome to triage this as accepted. Please also explore whether you can put new content within three of the existing sections: and, that way, avoid adding a new section to the concept list - we already have quite a few. of course, it's also fine to add new Task or Reference pages that cover multi-tenancy aspects.
/language en
/triage accepted
See also
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/wg multi-tenancy
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg multi-tenancy Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg multitenancy


Question: {{ if ( .Get "category" ) }}

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: uses the wrong background color(s); it's but should be (Kubernetes blue). I think the orange color was copied from by mistake. Proposed Solution: Revise the shortcode. Consider making the color be set once and then reused (“”). Page to Update: You would see the change on We should use blue backgrounds whether or not an announcement banner is showing. <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--Additional Information: PR updated the shortcode. /area web-development /triage accepted /priority backlog If fixing this, consider also moving the iframe scroll-margin-top fix to be more general. See and
this is a nit against PR but fixing it isn't at all urgent.
/lifecycle frozen We should fix this eventually.
I would like to help with this. /assign
/assign


Question: <ins> --- title: PR wranglers content_type: concept weight: 20 --- <!-- overview --> SIG Docs [approvers](/docs/contribute/participating/roles-and-responsibilites/#approvers) take week-long shifts [managing pull requests](https://github.com/kubernetes/website/wiki/PR-Wranglers) for the repository. This section covers the duties of a PR wrangler. For more information on giving good reviews, see [Reviewing changes](/docs/contribute/review/). <!-- body --> ## Duties Each day in a week-long shift as PR Wrangler: - Triage and tag incoming issues daily. See [Triage and categorize issues](/docs/contribute/review/for-approvers/#triage-and-categorize-issues) for guidelines on how SIG Docs uses metadata. - Review [open pull requests](https://github.com/kubernetes/website/pulls) for quality and adherence to the [Style](/docs/contribute/style/style-guide/) and [Content](/docs/contribute/style/content-guide/) guides. - Start with the smallest PRs (`size/XS`) first, and end with the largest (`size/XXL`). Review as many PRs as you can. - Make sure PR contributors sign the [CLA](https://github.com/kubernetes/community/blob/master/CLA.md). - Use [this](https://github.com/zparnold/k8s-docs-pr-botherer) script to remind contributors that haven’t signed the CLA to do so. - Provide feedback on changes and ask for technical reviews from members of other SIGs. - Provide inline suggestions on the PR for the proposed content changes. - If you need to verify content, comment on the PR and request more details. - Assign relevant `sig/` label(s). - If needed, assign reviewers from the `reviewers:` block in the file's front matter. - Use the `/approve` comment to approve a PR for merging. Merge the PR when ready. - PRs should have a `/lgtm` comment from another member before merging. - Consider accepting technically accurate content that doesn't meet the [style guidelines](/docs/contribute/style/style-guide/). Open a new issue with the label `good first issue` to address style concerns. ### Helpful GitHub queries for wranglers The following queries are helpful when wrangling. After working through these queries, the remaining list of PRs to review is usually small. These queries exclude localization PRs. All queries are against the main branch except the last one. - [No CLA, not eligible to merge](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+label%3A%22cncf-cla%3A+no%22+-label%3Ado-not-merge+label%3Alanguage%2Fen): Remind the contributor to sign the CLA. If both the bot and a human have reminded them, close the PR and remind them that they can open it after signing the CLA. **Do not review PRs whose authors have not signed the CLA!** - [Needs LGTM](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+-label%3Algtm+): Lists PRs that need an LGTM from a member. If the PR needs technical review, loop in one of the reviewers suggested by the bot. If the content needs work, add suggestions and feedback in-line. - [Has LGTM, needs docs approval](https://github.com/kubernetes/website/pulls?q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+label%3Algtm): Lists PRs that need an `/approve` comment to merge. - [Quick Wins](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+base%3Amaster+-label%3A%22do-not-merge%2Fwork-in-progress%22+-label%3A%22do-not-merge%2Fhold%22+label%3A%22cncf-cla%3A+yes%22+label%3A%22size%2FXS%22+label%3A%22language%2Fen%22+): Lists PRs against the main branch with no clear blockers. (change "XS" in the size label as you work through the PRs [XS, S, M, L, XL, XXL]). - [Not against the main branch](https://github.com/kubernetes/website/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+-label%3Ado-not-merge+label%3Alanguage%2Fen+-base%3Amaster): If the PR is against a `dev-` branch, it's for an upcoming release. Assign the [docs release manager](https://github.com/kubernetes/sig-release/tree/master/release-team#kubernetes-release-team-roles) using: `/assign @<manager's_github-username>`. If the PR is against an old branch, help the author figure out whether it's targeted against the best branch. ### When to close Pull Requests Reviews and approvals are one tool to keep our PR queue short and current. Another tool is closure. Close PRs where: - The author hasn't signed the CLA for two weeks. Authors can reopen the PR after signing the CLA. This is a low-risk way to make sure nothing gets merged without a signed CLA. - The author has not responded to comments or feedback in 2 or more weeks. Don't be afraid to close pull requests. Contributors can easily reopen and resume works in progress. Often a closure notice is what spurs an author to resume and finish their contribution. To close a pull request, leave a `/close` comment on the PR. {{< note >}} The [`fejta-bot`](https://github.com/fejta-bot) bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them.  PR wranglers should close issues after 14-30 days of inactivity. {{< /note >}} </ins>  No newline at end of file

Umbrella issue: Take Advanced Contributing Be a PR Wrangler for a week topic and create a new page, Participating in SIG Docs PR Wranglers.
/assign /kind cleanup
/language en


Question: dependent resources, based on the delete permissions of the owner. This control prevents unauthorized users from delaying owner object deletion. <ins> {{< note >}} Cross-namespace owner references are disallowed by design. Namespaced dependents can specify cluster-scoped or namespaced owners. A namespaced owner **must** exist in the same namespace as the dependent. If it does not, the owner reference is treated as absent, and the dependent is subject to deletion once all owners are verified absent. Cluster-scoped dependents can only specify cluster-scoped owners. In v1.20+, if a cluster-scoped dependent specifies a namespaced kind as an owner, it is treated as having an unresolvable owner reference, and is not able to be garbage collected. In v1.20+, if the garbage collector detects an invalid cross-namespace `ownerReference`, or a cluster-scoped dependent with an `ownerReference` referencing a namespaced kind, a warning Event with a reason of `OwnerRefInvalidNamespace` and an `involvedObject` of the invalid dependent is reported. You can check for that kind of Event by running `kubectl get events -A --field-selector=reason=OwnerRefInvalidNamespace`. {{< /note >}} </ins> ## Ownership and finalizers When you tell Kubernetes to delete a resource, the API server allows the

The following information was dropped in and is critical to include in the garbage collection ownerReferences topic:
/triage accepted /sig api-machinery
/priority important-soon


Question: If you prefer not to use [kubectl drain](/docs/reference/generated/kubectl/kubectl-commands/#drain) (such as to avoid calling to an external command, or to get finer control over the pod <del> eviction process), you can also programmatically cause evictions using the eviction API. You should first be familiar with using [Kubernetes language clients](/docs/tasks/administer-cluster/access-cluster-api/#programmatic-access-to-the-api) to access the API. The eviction subresource of a Pod can be thought of as a kind of policy-controlled DELETE operation on the Pod itself. To attempt an eviction (more precisely: to attempt to *create* an Eviction), you POST an attempted operation. Here's an example: {{< tabs name="Eviction_example" >}} {{% tab name="policy/v1" %}} {{< note >}} `policy/v1` Eviction is available in v1.22+. Use `policy/v1beta1` with prior releases. {{< /note >}} ```json { "apiVersion": "policy/v1", "kind": "Eviction", "metadata": { "name": "quux", "namespace": "default" } } ``` {{% /tab %}} {{% tab name="policy/v1beta1" %}} {{< note >}} Deprecated in v1.22 in favor of `policy/v1` {{< /note >}} ```json { "apiVersion": "policy/v1beta1", "kind": "Eviction", "metadata": { "name": "quux", "namespace": "default" } } ``` {{% /tab %}} {{< /tabs >}} </del> <ins> eviction process), you can also programmatically cause evictions using the eviction API. </ins> <del> You can attempt an eviction using `curl`: ```bash curl -v -H 'Content-type: application/json' https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json ``` The API can respond in one of three ways: - If the eviction is granted, then the Pod is deleted as if you sent a `DELETE` request to the Pod's URL and received back `200 OK`. - If the current state of affairs wouldn't allow an eviction by the rules set forth in the budget, you get back `429 Too Many Requests`. This is typically used for generic rate limiting of *any* requests, but here we mean that this request isn't allowed *right now* but it may be allowed later. - If there is some kind of misconfiguration; for example multiple PodDisruptionBudgets that refer the same Pod, you get a `500 Internal Server Error` response. For a given eviction request, there are two cases: - There is no budget that matches this pod. In this case, the server always returns `200 OK`. - There is at least one budget. In this case, any of the three above responses may apply. ## Stuck evictions In some cases, an application may reach a broken state, one where unless you intervene the eviction API will never return anything other than 429 or 500. For example: this can happen if ReplicaSet is creating Pods for your application but the replacement Pods do not become `Ready`. You can also see similar symptoms if the last Pod evicted has a very long termination grace period. In this case, there are two potential solutions: - Abort or pause the automated operation. Investigate the reason for the stuck application, and restart the automation. - After a suitably long wait, `DELETE` the Pod from your cluster's control plane, instead of using the eviction API. Kubernetes does not specify what the behavior should be in this case; it is up to the application owners and cluster owners to establish an agreement on behavior in these cases. </del> <ins> For more information, see [API-initiated eviction](/docs/concepts/scheduling-eviction/api-eviction/). </ins> ## {{% heading "whatsnext" %}} * Follow steps to protect your application by [configuring a Pod Disruption Budget](/docs/tasks/run-application/configure-pdb/).

The content in for the Eviction API can be used to expand the conceptual information in . Proposed by /assign /sig node /sig docs /language en
cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Bad bot /remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Nooo /remove-lifecycle stale


Question: 在服务器日志中没有出现 RBAC 拒绝消息，就可以删除 ABAC 鉴权器。 <!-- <del> ## Permissive RBAC Permissions </del> <ins> ### Permissive RBAC Permissions </ins> You can replicate a permissive policy using RBAC role bindings. --> <del> ## 宽松的 RBAC 权限   {#permissive-rbac-permissions} </del> <ins> ### 宽松的 RBAC 权限   {#permissive-rbac-permissions} </ins> 你可以使用 RBAC 角色绑定在多个场合使用宽松的策略。

There is a translation bug: " --remove-extra-permissions" should be changed to " --remove-extra-subjects" There is a typographical bug: ,%E4%BB%8E%20ABAC%20%E5%8D%87%E7%BA%A7,-%E5%8E%9F%E6%9D%A5%E8%BF%90%E8%A1%8C%E8%BE%83 "" and " RBAC " should become subdirectory of " ABAC "
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/language zh /kind bug
/assign


Question: <del> --- reviewers: - mtaufen - dawnchen title: Reconfigure a Node's Kubelet in a Live Cluster content_type: task min-kubernetes-server-version: v1.11 weight: 280 --- <!-- overview --> {{< feature-state for_k8s_version="v1.22" state="deprecated" >}} {{< caution >}} The [Dynamic Kubelet Configuration](https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/281-dynamic-kubelet-configuration) feature is deprecated in 1.22 and removed in 1.24. Please switch to alternative means distributing configuration to the Nodes of your cluster. {{< /caution >}} [Dynamic Kubelet Configuration](https://github.com/kubernetes/enhancements/issues/281) allowed you to change the configuration of each {{< glossary_tooltip text="kubelet" term_id="kubelet" >}} in a running Kubernetes cluster, by deploying a {{< glossary_tooltip text="ConfigMap" term_id="configmap" >}} and configuring each {{< glossary_tooltip term_id="node" >}} to use it. Please find documentation on this feature in [earlier versions of documentation](https://v1-23.docs.kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/). ## Migrating from using Dynamic Kubelet Configuration There is no recommended replacement for this feature that works generically across various Kubernetes distributions. If you are using managed Kubernetes version, please consult with the vendor hosting Kubernetes for the best practices for customizing your Kubernetes. If you are using `kubeadm`, refer to [Configuring each kubelet in your cluster using kubeadm](/docs/setup/production-environment/tools/kubeadm/kubelet-integration/). In order to migrate off the Dynamic Kubelet Configuration feature, the alternative mechanism should be used to distribute kubelet configuration files. In order to apply configuration, config file must be updated and kubelet restarted. See the [Set Kubelet parameters via a config file](/docs/tasks/administer-cluster/kubelet-config-file/) for information. Please note, the `DynamicKubeletConfig` feature gate cannot be set on a kubelet starting v1.24 as it has no effect. However, the feature gate is not removed from the API server or the controller manager before v1.26. This is designed for the control plane to support nodes with older versions of kubelets and for satisfying the [Kubernetes version skew policy](/releases/version-skew-policy/). </del>

This is a Cleanup Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be removed Remove Why is this needed v1.24 of Kubernetes will remove the dynamic kubelet config feature (deprecated since Kubernetes v1.22) Comments /sig node /language en /milestone 1.24
/triage accepted
See
/assign
~BTW Any PR for this should target the branch dev-1.24~ That advice is now stale. Kubernetes v1.24 has shipped.
Okay... But I want to clear that you want me to remove the entire page from "Administer a Cluster" . Am I Right ?
/assign
This won't be in 1.24's docs, clearing the milestone. This can be to subsequent milestones if needed. (or closed if I'm missing a PR somewhere) /milestone clear
We should have updated the page to state when the removal is due, or to remove the commitment if we're no longer sure.
/label good-first-issue All you need to do is remove from the docs.
The label(s) /label good-first-issue api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactorlabels -additionallabelslabels -? <detailsIn response to : /label good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue states that the feature was removed. why did we take this change out of the v1.24 milestone?
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command.

<detailsIn response to : /good-first-issue states that the feature was removed. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should I also remove all the links that lead to ? for example, here:
/remove-milestone 1.24
Not required, but much better if you do.
hey, I have made the required changes in the PR, could you please review it?
can i work on this issue as well?
You're welcome to. The next step on contributing is to review
will do !
hey , this is my contribution and i would be glad to help and have an insight. can you tell me what to do? im new, just trying to get familiar with all of this
How to work on this: Please open a PR that removes If there are any other English pages that link to then please turn the links into plain text, or otherwise fix them up.
/assign
The issue has been resolved. all are removed or commented out. still there are some mention of this like but they there only to point out that the dynamic kubelet config is removed from v1.24.
I don't see any updates from your side; so unassigning you. Please feel free to assign, if you come back here again and are willing to work on /unassign
i think this issue is resolved.
If is addressed somewhere in the PR then can you please share a link of the respective PR? Thanks!
Can I work on this issue is this still open ?
heyyo can you please confirm what branch this issue fix needs to be targeted on? Thanks in advance:)
To remove that page, make the change to the main branch. Ideally, also find any English pages - just English - that link to and edit them so that they don't.
/assign
Hi This is my first contribution to open source, I have deleted the requested page. Should I proceed to open a pull request?
heyyo if you don't mind I was already working on the issue and have made the changes three days ago just haven't pushed them yet
of course! I noticed you did not claim the issue with the bot so I thought I was good to go :)
/assign
update : will create a PR tomorrow.

/assign
is this issue still open ??
Hi, I want to work on this. Is this stll open ?
/assign
Hi. There's a white-on-green “Open” marker at the left of the top navigation, at least in the default desktop view. That indicates that the issue is open. Anyone is welcome to suggest a fix, or to propose closing the issue if the fix has already happened. I can see that the page is still there at the time of writing.
hi I can fix this issue u can assign me it.
/assign
where are the html file of this page in the repo


Question: kubectl delete svc nginx ``` <ins> Delete the persistent storage media for the PersistentVolumes used in this tutorial. ```shell kubectl get pvc ``` ``` NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE www-web-0   Bound    pvc-2bf00408-d366-4a12-bad0-1869c65d0bee   1Gi        RWO            standard       25m www-web-1   Bound    pvc-ba3bfe9c-413e-4b95-a2c0-3ea8a54dbab4   1Gi        RWO            standard       24m www-web-2   Bound    pvc-cba6cfa6-3a47-486b-a138-db5930207eaf   1Gi        RWO            standard       15m www-web-3   Bound    pvc-0c04d7f0-787a-4977-8da3-d9d3a6d8d752   1Gi        RWO            standard       15m www-web-4   Bound    pvc-b2c73489-e70b-4a4e-9ec1-9eab439aa43e   1Gi        RWO            standard       14m ``` ```shell kubectl get pv ``` ``` NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE pvc-0c04d7f0-787a-4977-8da3-d9d3a6d8d752   1Gi        RWO            Delete           Bound    default/www-web-3   standard                15m pvc-2bf00408-d366-4a12-bad0-1869c65d0bee   1Gi        RWO            Delete           Bound    default/www-web-0   standard                25m pvc-b2c73489-e70b-4a4e-9ec1-9eab439aa43e   1Gi        RWO            Delete           Bound    default/www-web-4   standard                14m pvc-ba3bfe9c-413e-4b95-a2c0-3ea8a54dbab4   1Gi        RWO            Delete           Bound    default/www-web-1   standard                24m pvc-cba6cfa6-3a47-486b-a138-db5930207eaf   1Gi        RWO            Delete           Bound    default/www-web-2   standard                15m ``` ```shell kubectl delete pvc www-web-0 www-web-1 www-web-2 www-web-3 www-web-4 ``` ``` persistentvolumeclaim "www-web-0" deleted persistentvolumeclaim "www-web-1" deleted persistentvolumeclaim "www-web-2" deleted persistentvolumeclaim "www-web-3" deleted persistentvolumeclaim "www-web-4" deleted ``` ```shell kubectl get pvc ``` ``` No resources found in default namespace. ``` </ins> {{< note >}} You also need to delete the persistent storage media for the PersistentVolumes used in this tutorial. Follow the necessary steps, based on your environment, storage configuration, and provisioning method, to ensure that all storage is reclaimed. <del> {{< /note >}} </del> <ins> {{< /note >}} </ins>  No newline at end of file

Hello Doc Team, This document is very detailed to explain the steps related to Statefulsets. Thank you, also I would like to suggest set of cleanup steps to delete the volumes assigned as part of the tutorial. This would help to leave the system clean at the end of tutorial. {code} As part of clean up run the following commands to check the physical volumes assigned. kubectl get pvc kubectl get pv #delete all the volumes assigned. kubectl delete pvc www-web-0 www-web-1 www-web-2 www-web-3 www-web-4 #validate the volumes released. kubectl get pvc {code} Thank you, Balraj
/kind feature /language en
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
There is a note in that which refers to clear all the persistent volume we have created . so is this good idea to add the clean up step also?
I agree. Adding a Clean Up section to that document is a good idea. /triage accepted /priority backlog
I would like to work on this. /assign
/sig storage
i have create a PR for the same. PTAL
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
i have opened the PR and addressed all the comments could you please take a look .


Question: serving the health check at `/healthz`. You can test this: ```shell <del> kubectl get pod -o wide -l run=source-ip-app </del> <ins> kubectl get pod -o wide -l app=source-ip-app </ins> ``` The output is similar to this: ```

On page: Based on the labels I see in the pod, I believe that the line that says: should be changed to:
/assign
/triage accepted
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/language en


Question: ### Migrating to the `systemd` driver in kubeadm managed clusters <del> Follow this [Migration guide](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver) </del> <ins> Follow this [Migration guide](/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/) </ins> if you wish to migrate to the `systemd` cgroup driver in existing kubeadm managed clusters. ## Container runtimes

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: In there is a table broken with GitHub Markdown renderer. ! BTW, the table is not broken with Hugo Markdown renderer. ! Proposed Solution: Page to Update: No Page on to Update. Just a Markdown file on GitHub to update. <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--/assign
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We support viewing the rendered documentation via I don't think this issue is valid.
/priority awaiting-more-evidence /langauge en
Thank for your comment. I will describe a bit more about this issue I opened. [As-is] Yes, although the second row of the Markdown code has 4 columns and other rows have 3 columns, this table is rendered alright by Hugo Markdown renderer. This is a issue about improving consistency of the number of columns in the Markdown code, regardless of the type of Markdown renderer. [To-be]

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: In , the link is broken. Proposed Solution: In Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--/assign
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
already logged here: /close /sig cluster-lifecycle
Closing this issue. <detailsIn response to : already logged here: /sig cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ``` 1. Perintah `kubectl create service` membuat konfigurasi untuk objek Service dan menyimpannya di `/tmp/srv.yaml`. <del> 1. Perintah `kubectl create --edit` membuka file konfigurasi untuk diedit sebelum objek dibuat. </del> <ins> 1. Perintah `kubectl create --edit` membuka berkas konfigurasi untuk disunting sebelum objek dibuat. </ins>

This issue is intended for contributor workshop as part of Cloud Native Day Indonesia on 7th Nov 2020. Please don't take it if you are not part of the workshop. There are several old translated pages that contain the english glossary "file", supposed to use Indonesian "berkas" as per KBBI. Please claim the task to fix them by giving comments on this Github issue. /language id Tasks to fix "file" -"berkas": [x] docs/tasks/manage-kubernetes-objects/imperative-[x] docs/tasks/manage-kubernetes-objects/declarative-[ ] docs/tasks/manage-kubernetes-objects/imperative-[ ] docs/tasks/configure-pod-container/configure-pod-[ ] docs/tasks/administer-cluster/sysctl-[ ] [x] [ ] docs/concepts/storage/storage-[ ] docs/concepts/services-networking/connect-applications-[ ] docs/concepts/services-[x] [x] docs/concepts/cluster-
/triage accepted
/assign
I'm taking this page docs/tasks/manage-kubernetes-objects/imperative-
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'm taking this page docs/tasks/manage-kubernetes-objects/declarative-
may i change this file:
thanks feel free to raise the PR and link this issue
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I take this page :
please let me change another part. I will try to change this file
sure just mention this issue number when you raise the pull request /reopen
Reopened this issue. <detailsIn response to : please let me change another part sure just mention this issue number when you raise the pull request Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Limiting port ranges of communication | This recommendation may be a bit self-explanatory, but wherever possible you should only expose the ports on your service that are absolutely essential for communication or metric gathering. | 3rd Party Dependency Security | It is a good practice to regularly scan your application's third party libraries for known security vulnerabilities. Each programming language has a tool for performing this check automatically. | Static Code Analysis | Most languages provide a way for a snippet of code to be analyzed for any potentially unsafe coding practices. Whenever possible you should perform checks using automated tooling that can scan codebases for common security errors. Some of the tools can be found at: https://owasp.org/www-community/Source_Code_Analysis_Tools | <del> Dynamic probing attacks | There are a few automated tools that you can run against your service to try some of the well known service attacks. These include SQL injection, CSRF, and XSS. One of the most popular dynamic analysis tools is the [OWASP Zed Attack proxy](https://owasp.org/www-project-zap/) tool. | </del> <ins> Dynamic probing attacks | There are a few automated tools that you can run against your service to try some of the well known service attacks. These include SQL injection, CSRF, and XSS. One of the most popular dynamic analysis tools is the [OWASP Zed Attack proxy](https://www.zaproxy.org/) tool. | </ins> {{< /table >}}

This is a Bug Report <!-- Thanks for filing an issue! Before submitting, please fill in the following information. --<!-- See for guidance on writing an actionable issue description. --<!--Required Information--Problem: The link to in the code security section of leads to a 404 error. Proposed Solution: Replace the link with a working one or remove it. Page to Update: <!--Optional Information (remove the comment tags around information you would like to include)--<!--Kubernetes Version:--<!--Additional Information:--
/sig security /triage accepted
can we use the following link for ZAP: The above link redirects to the ZAP website.
cc your thoughts on this
/assign


Question: description: Browse terminology, command line syntax, API resource types, and setup tool documentation. button: View Reference button_path: /docs/reference <del> - name: training title: "Training" description: "Get certified in Kubernetes and make your cloud native projects successful!" button: "View training" button_path: "/training" </del> - name: contribute title: Contribute to Kubernetes description: Anyone can contribute, whether you're new to the project or you've been around a long time. button: Find out how to help button_path: /docs/contribute <ins> - name: training title: "Training" description: "Get certified in Kubernetes and make your cloud native projects successful!" button: "View training" button_path: "/training" </ins> - name: Download title: Download Kubernetes description: Install Kubernetes or upgrade to the newest version.

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be /enhanced? As it can be clearly seen at that the layout of this page is looking somewhat inconsistent due to this visible big-small difference in 'Cards' at 2nd and 3rd row. ! ! Solution: I think this issue can be easily resolved by replacing the 'Card' of (3rd Column, 2nd Row) with (1st Row, 3rd Column) like this:- ! ! As 'Cards' of 2nd row are all vertically tall and 'Cards' of 3rd row are all vertically small. So, replacing these 2nd 'Cards' with each other will possibly solve the problem. By Card, i mean that whole thing inside yellow line (Heading, description, hyperlinks and button)
/area web-development
/assign
A was previously filed. See and for more context.
/retitle Poor alignment for tiles and buttons on home page for docs
/triage accepted /priority backlog
/assign
/retitle Poor alignment for tiles on home page for docs


Question: description: Browse terminology, command line syntax, API resource types, and setup tool documentation. button: View Reference button_path: /docs/reference <del> - name: training title: "Training" description: "Get certified in Kubernetes and make your cloud native projects successful!" button: "View training" button_path: "/training" </del> - name: contribute title: Contribute to Kubernetes description: Anyone can contribute, whether you're new to the project or you've been around a long time. button: Find out how to help button_path: /docs/contribute <ins> - name: training title: "Training" description: "Get certified in Kubernetes and make your cloud native projects successful!" button: "View training" button_path: "/training" </ins> - name: Download title: Download Kubernetes description: Install Kubernetes or upgrade to the newest version.

This is a Feature Request <!-- Please only use this template for submitting feature/enhancement requests --<!-- See for guidance on writing an actionable issue description. --What would you like to be /enhanced? As it can be clearly seen at that the layout of this page is looking somewhat inconsistent due to this visible big-small difference in 'Cards' at 2nd and 3rd row. ! ! Solution: I think this issue can be easily resolved by replacing the 'Card' of (3rd Column, 2nd Row) with (1st Row, 3rd Column) like this:- ! ! As 'Cards' of 2nd row are all vertically tall and 'Cards' of 3rd row are all vertically small. So, replacing these 2nd 'Cards' with each other will possibly solve the problem. By Card, i mean that whole thing inside yellow line (Heading, description, hyperlinks and button)
/area web-development
/assign
A was previously filed. See and for more context.
/retitle Poor alignment for tiles and buttons on home page for docs
/triage accepted /priority backlog
/assign
/retitle Poor alignment for tiles on home page for docs


Question: --> ## 将 ConfigMap 数据添加到一个卷中 <del> 如基于文件创建 ConfigMap](#create-configmaps-from-files) 中所述，当你使用 </del> <ins> 如基于文件创建 [ConfigMap](#create-configmaps-from-files) 中所述，当你使用 </ins> `--from-file` 创建 ConfigMap 时，文件名成为存储在 ConfigMap 的 `data` 部分中的键， 文件内容成为键对应的值。

! ‘ ConfigMap](#create-configmaps-from-files) ’
This issue is currently awaiting triage. SIG Docs takes a lead on issue triage for this website, but any Kubernetes member can accept issues by applying the label. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: 1. Install the [Google Cloud SDK](https://cloud.google.com/sdk/). <del> 2. Run the `kubectl` installation command: </del> <ins> 1. Run the `kubectl` installation command: </ins> ```shell gcloud components install kubectl ``` <del> 3. Test to ensure the version you installed is up-to-date: </del> <ins> 1. Test to ensure the version you installed is up-to-date: </ins> ```shell kubectl version --client

There are no details or instructions on how to validate the binary downloaded from this page via some form of checksum, sha256 or otherwise. For automation purposes, eg. ansible playbook, being able to validate any binary downloads is a requirement of project stakeholders.
The site doesn't allow browsing, but the file is there as kubectl.sha256 afterall. Might be worth mentioning this in the document?
/retitle Document how to validate kubectl binary against checksum / signature /language en /kind feature /triage accepted The page in question is
To fix this: update to use download links from for example also explain to readers that checksum files are available that they can use to compare binaries obtained from elsewhere for example The and shortcodes are useful for automatically filling in the current Kubernetes version. /help /sig cli
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : To fix this: update to use download links from explain to readers that checksum files are available that they can use to compare binaries obtained from elsewhere /sig cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I may be able to help with this /assign