Question: resourceName     string resourceQuantity resource.Quantity devs             checkpoint.DevicesPerNUMA <ins> topology         bool </ins> } func TestPodContainerDeviceAllocation(t *testing.T) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: id: monitoring-grafana port: 80 containerPort: 80 <ins> labels: name: grafana kubernetes.io/cluster-service: "true" </ins> selector: name: influxGrafana

As per we should add a common label to all of our add-on pods: DNS, logging, monitoring, etc. The proposal is to add "": "true" for now.
FWIW "true" has to be quoted in YAML
Can you close this one once it's completed.


Question: <ins> # Kubernetes Design Overview Kubernetes build on top of [Docker](http://www.docker.io) to construct a clustered container scheduling service.  The goals of the project are to enable users to ask a Kubernetes cluster to run a set of containers.  The system will automatically pick a worker node to run those containers on. As container based applications and systems get larger, some tools are provided to facilitate sanity. This includes ways for containers to find and communicate with each other and ways to work with and manage sets of containers that do similar work. When looking at the arechitecture of the system, we'll break it down to services that run on the worker node and services that play a "master" role. ## Key Concept: Container Pod While Docker itself works with individual containers, Kubernetes works with a `pod`.  A `pod` is a group of containers that are scheduled onto the same physical node.  In addition to defining the containers that run in the pod, the containers in the pod all use the same network namespace/IP and define a set of storage volumes.  Ports are also mapped on a per-pod basis. ## The Kubernetes Node The Kubernetes node has the services necessary to run Docker containers and be managed from the master systems. The Kubernetes node design is an extension of the [Container-optimized Google Compute Engine image](https://developers.google.com/compute/docs/containers#container-optimized_google_compute_engine_images).  Over time these plan is for these images/nodes to merge and be the same thing used in different ways. It has the services necessary to run Docker containers and be managed from the master systems. Each node runs Docker, of course.  Docker takes care of the details of downloading images and running containers. ### Kubelet The second component on the node called the `kubelet`.  The Kubelet is the logical successor (and rewrite in go) of the [Container Agent](https://github.com/GoogleCloudPlatform/container-agent) that is part of the Compute Engine image. The Kubelet works in terms of a container manifest.  A container manifest (defined [here](https://developers.google.com/compute/docs/containers#container_manifest)) is a YAML file that describes a `pod`.  The Kubelet takes a set of manifests that are provided in various mechanisms and ensures that the containers described in those manifests are started and continue running. There are 4 ways that a container manifest can be provided to the Kubelet: * **File** Path passed as a flag on the command line.  This file is rechecked every 20 seconds (configurable with a flag). * **HTTP endpoint** HTTP endpoint passed as a parameter on the command line.  This endpoint is checked every 20 seconds (also configurable with a flag.) * **etcd server**  The Kubelet will reach out and do a `watch` on an [etcd](https://github.com/coreos/etcd) server.  The etcd path that is watched is `/registry/hosts/$(hostname -f)`.  As this is a watch, changes are noticed and acted upon very quickly. * **HTTP server** The kubelet can also listen for HTTP and respond to a simple API (underspec'd currently) to submit a new manifest. ### Kubernetes Proxy Each node also runs a simple network proxy.  This reflects `services` as defined in the Kubernetes API on each node and can do simple TCP stream forwarding or round robin TCP forwarding across a set of backends. ## The Kubernetes Master The Kubernetes master is split into a set of components.  These work together to provide an unified view of the cluster. ### etcd All persistent master state is stored in an instance of `etcd`.  This provides a great way to store configuration data reliably.  With `watch` support, coordinating components can be notified very quickly of changes. ### Kubernetes API Server This server serves up the main [Kubernetes API](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/api). It validates and configures data for 3 types of objects: * `pod`: Each `pod` has a representation at the Kubernetes API level. * `service`: A service is a configuration unit for the proxies that run on every worker node.  It is named and points to one or more Pods. * `replicationController`: A replication controller takes a template and ensures that there is a specified number of "replicas" of that template running at any one time.  If there are too many, it'll start more.  If there are too few, it'll kill some. Beyond just servicing REST operations, validating them and storing them in `etcd`, the API Server does two other things: * Schedules pods to worker nodes.  Right now the scheduler is very simple. * Synchronize pod information (where they are, what ports they are exposing) with the service configuration. ### Kubernetes Controller Manager Server The `repliationController` type described above isn't strictly necessary for Kubernetes to be useful.  It is really a service that is layered on top of the simple `pod` API.  To enforce this layering, the logic for the repliationController is actually broken out into another server.  This server watches `etcd` for changes to `replicationController` objects and then uses the public Kubernetes API to implement the repliation algorithm. ### Key Concept: Labels Pods are organized using labels.  Each pod can have a set of key/value labels set on it. Via a "label query" the user can identify a set of `pods`.  This simple mechanism is a key part of how both `services` and `replicationControllers` work.  The set of pods that a `service` points at is defined with a label query.  Similarly the population of pods that a `replicationController` is monitoring is also defined with a label query. Label queries would typically be used to identify and group pods into, say, a tier in an application.  You could also idenitfy the stack such as `dev`, `staging` or `production`. These sets could be overlapping.  For instance, a service might point to all pods with `tier in (frontend), stack in (prod)`.  Now say you have 10 replicated pods that make up this tier.  But you want to be able to 'canary' a new version of this component.  You could set up a `replicationController` (with `replicas` set to 9) for the bulk of the replicas with labels `tier=frontend,stack=prod,canary=no` and another `replicationController` (with `replicas` set to 1) for the canary with labels `tier=frontend, stack=prod, canary=yes`.  Now the service is covering both the canary and non-canary pods.  But you can mess with the `replicationControllers` separately to test things out, monitor the results, etc. ## Network Model Kubernetes expands the default Docker networking model.  The goal is to have each `pod` have an IP in a shared networking namespace that has full communication with other physical computers and containers across the network.  In this way, it becomes much less necessary to map ports. For the Google Compute Engine cluster configuration scripts, [advanced routing](https://developers.google.com/compute/docs/networking#routing) is set up so that each VM has a extra 256 IP addresses that get routed to it.  This is in addition to the 'main' IP address assigned to the VM that is NAT-ed for Internet access.  The networking bridge (called `cbr0` to differentiate it from `docker0`) is set up outside of Docker proper and only does NAT for egress network traffic that isn't aimed at the virtual network. Ports mapped in from the 'main IP' (and hence the internet if the right firewall rules are set up) are proxied in user mode by Docker.  In the future, this should be done with `iptables` by either the Kubelet or Docker: [Issue #15](https://github.com/GoogleCloudPlatform/kubernetes/issues/15). ## Release Process Right now "building" or "releasing" Kubernetes consists of some scripts (in `release/` to create a `tar` of the necessary data and then uploading it to Google Cloud Storage.  In the future we will generate Docker images for the bulk of the above described components: [Issue #19](https://github.com/GoogleCloudPlatform/kubernetes/issues/19). ## GCE Cluster Configuration The scripts and data in the `cluster/` directory automates creating a set of Google Compute Engine VMs and installing all of the Kubernetes components.  There is a single master node and a set of worker (called minion) nodes. `config-default.sh` has a set of tweakable definitions/parameters for the cluster. The heavy lifting of configuring the VMs is done by [SaltStack](http://www.saltstack.com/). The bootstrapping works like this: 1. The `kube-up.sh` script uses the GCE [`startup-script`](https://developers.google.com/compute/docs/howtos/startupscript) mechanism for both the master node and the minion nodes. * For the minion, this simply configures and installs SaltStack.  The network range that this minion is assigned is baked into the startup-script for that minion. * For the master, the release files are downloaded from GCS and unpacked.  Various parts (specifically the SaltStack configuration) are installed in the right places. 2. SaltStack then installs the necessary servers on each node. * All go code is currently downloaded to each machine and compiled at install time. * The custom networking bridge is configured on each minion before Docker is installed. * Configuration (like telling the `apiserver` the hostnames of the minions) is dynamically created during the saltstack install. 3. After the VMs are started, the `kube-up.sh` script will call `curl` every 2 seconds until the `apiserver` starts responding. `kube-down.sh` can be used to tear the entire cluster down.  If you build a new release and want to update your cluster, you can use `kube-push.sh` to update and apply (`highstate` in salt parlance) the salt config. ### Cluster Security As there is no security currently built into the `apiserver`, the salt configuration will install `nginx`.  `nginx` is configured to serve HTTPS with a self signed certificate.  HTTP basic auth is used from the client to `nginx`.  `nginx` then forwards the request on to the `apiserver` over plain old HTTP. The password is generated randomly as part of the `kube-up.sh` script and stored in `~/.kubernetes_auth`. </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to endChanges 5/12/2021 gmsa CRD is using v1beta1 apis started fix: /sig windows /triage accepted
Still failing in 1.18 - Is this expected?

We should have a good overarching technical overview of the design of kubernetes. For new developers or those that want to understand a system deeply before using it. This should lay out the major moving parts and how they work together. says:


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-Scheduler exposes metrics in a secure manner (TLS and authentication required) on port by default. If one does not authenticate itself, the error message displayed is: What you expected to happen: Kube-Scheduler should emit a clear and understandable error message if authentication fails. Kube-Controller-Manager does that the following way: How to reproduce it (as minimally and precisely as possible): up a Kubernetes Cluster to access the exposed metrics endpoint (https://metrics-server:/metrics) without authentication Anything else we need to know?: xref: Environment: Kubernetes version (use ): 1.18.6 Cloud provider or hardware configuration: Bare metal OS (e.g: ): Debian GNU/Linux 10 (buster) Kernel (e.g. ): 4.19.0-9-amd64 Install tools: Kubeadm (Kube-Scheduler )
/sig scheduling instrumentation
If ok, I can try this fix /assign
/assign


Question: return d } <ins> // helper to create RS with given availableReplicas func newRSWithAvailable(name string, specReplicas, statusReplicas, availableReplicas int) *extensions.ReplicaSet { rs := rs(name, specReplicas, nil, metav1.Time{}) rs.Status = extensions.ReplicaSetStatus{ Replicas:          int32(statusReplicas), AvailableReplicas: int32(availableReplicas), } return rs } </ins> func TestRequeueStuckDeployment(t *testing.T) { pds := int32(60) failed := []extensions.DeploymentCondition{

Please see There are no unit tests currently for that method. This would be a good started PR for someone interested in helping with Kubernetes.
/sig apps
/cc
/for-new-contributors
/assign
I want this done by the end of this week so we can cherry-pick both the unit test and back to 1.7 and potentially 1.6.
if nobody else is working on it, i'll look into this.
Hey Please do so and cc me on your PR, I haven't had any time to start on it.
/priority important-soon


Question: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } <del> case "PUT": </del> <ins> case "PATCH": </ins> switch req.URL.Path { case "/namespaces/test/pods/cassandra": return &http.Response{StatusCode: 200, Body: objBody(codec, &pods.Items[0])}, nil

and work in this pattern: first issuing a "GET" to the API server to retrieve the latest object, then adding the label/annotation to the object, and finally issuing an "PUT" with the modified object. The request will fail if the object stored in the server has been updated between the "GET" and "PUT", even if the update has no conflict with the label/annotate request. In such cases, instead of returning an error, should retry without disturbing the end user. There are two ways to implement this: in kubectl: simply retry the label/annotate process several times if the server rejected the request because the object has been updated. The only exception is when the is set, then kubectl shouldn't retry. the strategicMergePatch instead of calling Update at client-side. PATCH is more robust because PATCH does the "get, modify, update" in the server-side, so there is less chance to have a conflict, and PATCH will retry at the server-side (see ). We need to make some adaptions though: kubectl annotate/label has the , when it's false, kubectl should retrieve the latest object to confirm there is no existing annotation/label of the same key, and then proceed with issuing the patch. if the is set, [UPDATE:] we just need to specify the resourVersion in the patch, then it will be treated as a precondition by the UpdateResource() handler in the API server. I plan to take the first approach as an immediate fix.
Immediate need is to fix the flaky e2e test AIUI.
e2e? do we have an issue out there? is an immediate fix to the flake in test-
If this is for 1.1, it needs to be P0 at this point. Should it be?
Fwiw I use annotations both from cmd line as an admin and from scripts, to experiment with loadbalancer settings. In the case of the former, I've never really had a collision, and in the case of the latter my scripts know how to retry like we retry all other updates.
Thank you When used in scripts, the annotate and label commands have collisions frequently, e.g. in the test-, see . IMO, we shouldn't require users to retry in their scripts if we know there is a simple way to avoid the collisions. Anyway, and are sent to fix this.


Question: // returned, along with a response header containing the list of protocols the // server can accept. func Handshake(req *http.Request, w http.ResponseWriter, serverProtocols []string) (string, error) { <del> clientProtocols := req.Header[http.CanonicalHeaderKey(HeaderProtocolVersion)] </del> <ins> clientProtocols := commaSeparatedHeaderValues(req.Header[http.CanonicalHeaderKey(HeaderProtocolVersion)]) </ins> if len(clientProtocols) == 0 { return "", fmt.Errorf("unable to upgrade: %s is required", HeaderProtocolVersion) }

What happened: I'm trying to establish a SPDY connection to exec into a pod. I sent this header: X-Stream-Protocol-Version: , , , The request failed with Forbidden - unable to upgrade: unable to negotiate protocol: client supports [, , , ], server accepts [ ] It appears that the server is not parsing the list out of the HTTP header and is treating it as a single value. What you expected to happen: Protocol negotiation should succeed. RFC2616 says: I can't simply send a multiple headers, because the HTTP library I'm using has no mechanism for sending separate headers as opposed to combining them. Kubernetes should parse the header whether it's given as separate headers or a list. Anything else we need to know?: I'm having to use SPDY because the web sockets implementation of exec . Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure VMOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux AdamArisDev 4.15.0-1063-azure -Ubuntu SMP Fri Nov 8 09:30:20 UTC 2019 x8664 x8664 x8664 GNU/LinuxOthers: My cluster is running in minikube v1.1.0.

/sig api-machinery
Thinking of the following change:
Since none of the accepted protocols include a comma, we should probably just iterate over all the headers, split by , trim each resulting item, omitting empty strings when computing clientProtocols, regardless of the number of headers.
Iterate over
Thanks for the quick update. I don't know what the expected HTTP behavior is when you use multiple headers and also have lists in each header, but if the same rule applies that multiple headers should be treated the same as one header with the items separated by commas, then presumably: Foo: a, b Foo: c, d should be the same as Foo: a, b, c, d in which case suggestion of basically moving the split into the inner loop would be more correct. In any case, I suppose a bug fix of this nature would only affect newer versions of Kubernetes? (I don't know if you do bug-fix-only patches to older versions.) I guess as a workaround I could try to catch the Forbidden error, look for the "unable to negotiate protocol" text, and switch to using only a single header value for older Kubernetes versions. Or is there a better approach?
That said, presumably this could also be handled in a lower layer - in the HTTP layer where headers are parsed - which might fix similar bugs with other headers that might exist elsewhere. (I think the only general exception to the equivalency rule is the Set-Cookie header because in practice cookies can contain commas but aren't quoted.)
That's correct. Given all the currently supported protocols are available in all supported kubernetes versions (the most recent protocols were in in v1.2.0 and in in v1.4.0), I'd probably just pick the specific protocol you want. If you wanted to detect a protocol failure, the X-Accepted-Stream-Protocol-Versions header is sent on a forbidden response due to protocol negotiation failure.
That's good to know. I can just stick to v4, then, since I don't expect to need anything that old. :-) Thanks.
/reopen I agree, should this fix be more general? Some other http headers have also the comma problems.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I would not recommend that, as not all headers value handling can safely assume commas are value delimiters.
I would keep this issue scoped to this particular header
/assign (since you are already working on this one, thanks!)
/close fixed in
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } stopCh := make(chan struct{}) // Get the completed config cc := c.Complete()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: err := wait.Poll(policyCachePollInterval, policyCachePollTimeout, func() (bool, error) { response, err := c.SubjectAccessReviews().Create(review) <del> // GKE doesn't enable the SAR endpoint.  Without this endpoint, we cannot determine if the policy engine // has adjusted as expected.  In this case, simply wait one second and hope it's up to date // TODO: Should have a check for the provider here but that introduces too tight of // coupling with the `framework` package. See: https://github.com/kubernetes/kubernetes/issues/76726 if apierrors.IsNotFound(err) { logf("SubjectAccessReview endpoint is missing") time.Sleep(1 * time.Second) return true, nil } </del> if err != nil { return false, err }

What would you like to be : The function in should check for GKE and allow a missing SubjectAccessReview endpoint. If we get a 404 on another provider an error should still be returned. GKE doesn't enable the SAR endpoint. Without this endpoint, we cannot determine if the policy engine has adjusted as expected. In this case, we currently wait one second and hope it's up to date without returning an error and that it has adjusted as expected. Why is this needed: A 404 on that endpoint is not expected behavior for any provider other than GKE so it doesn't make any sense to implicitly allow 404s and hope that everything is ok. If it is a special, allowable, condition for GKE we should make that explicit in the code. The problem with doing it now is that checking the provider requires access to the TestContext which forces us to import the entire package. This goes against the goal of At the same time, we don't want to cause every test/caller of this to have to (1) know that GKE acts differently in this situation and code that one special path. This may be the lesser of the evils though, but regardless of how it is accomplished, at some point, the code should be validating this flow. Open to other solutions to this; maybe a specific test to test this code path (in which case this helper can be assumed to not be responsible for it?) Not sure if that is considered sufficient as it creates an awkward coupling between a helper function's assumptions and the existence of particular test.
/sig testing /area test /area test-framework
I think that comment no longer applies, and we can just remove that GKE special case. IIRC, GKE didn't initially enable serving the group, but it does now.
OK; I will: confirm GKE handling paste link here for history Put up PR removing the special case entirely


Question: serviceAccountName: csi-provisioner containers: - name: csi-provisioner <del> image: quay.io/k8scsi/csi-provisioner:v1.0.1 </del> <ins> image: quay.io/k8scsi/csi-provisioner:v1.1.0-rc2 </ins> args: - "--provisioner=csi-hostpath" - "--csi-address=$(ADDRESS)"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-gce Which test(s) are failing: Since when has it been failing: unknown Testgrid link: Reason for failure: unknown Anything else we need to know: PR that introduces the new test: /assign /sig storage /kind flake /priority critical-urgent the job is PR blocking and is flaking mutliple times in a row.
/remove-kind bug
from SIG Testing mentioned that probably the whole job is timing out, so this might not be a SIG Storage problem per-se. we need the owners to triage the problem.
Here's one example run:
i've sent an email to k-dev to see who reacts. increasing the timeout is the preferred option for now.
To clarify I think this is a real failure that really happened at least twice, but it is likely not the primary cause of pull-kubernetes-e2e-gce currently being broken, and almost all failed pull-kubernetes-e2e-gce runs that I see had this test pass.
Let's use this bug to triage this specific test case
The first provision succeeded: 0404 18:32:48. 1 ] provision "multivolume-4799/pvc-mvjvq" class "multivolume-4799-csi-hostpath-multivolume-4799-scjmgx4": started The second one failed because it couldn't find the storage class: I0404 18:32:50. 1 ] provision "multivolume-4799/pvc-985xh" class "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq": started E0404 18:32:50. 1 ] provision "multivolume-4799/pvc-985xh" class "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq": error getting claim's StorageClass's fields: storageClass "multivolume-4799-csi-hostpath-multivolume-4799-scm9tnq" not found
The provisioner doesn't retry/requeue if the storageclass is not found.

I guess it is flaky because the pvc is created immediately after the sc and even though the provisioner has watches on both, it's possible it sees pvc before sc? It's a bug in the external library, not an issue with in-tree because it retries everything every 15s. So I'll need to update the library, update hostpath plugin, then update our manifests here. btw, how did you find that log
There is probably something else wrong too* because every other dynamic provisioning test creates pvc immediately after the sc. Why does only this one flake?
We store all the CSI driver logs for each test case under artifacts
Should we just add wait before and ? Or are there any ways to ensure that storageClass is ready? (like checking that storageClass can be got after creation.)
I don't think wait is the solution. Even if the test does a Get after Create, there is no guarantee that the provisioner pod's informer has seen the StorageClass too. I've opened for now I am still unsure why only this test is flaking frequently, if the issue is race between controller seeing PVC first or SC first, one would expect a lot of other external provisioner tests to fail too...
I guess given the frequency of failures, waiting for SC to exist after the Create call could be a good stopgap solution. Fixing the lib-hostpath plugin-test will take a lot more time.
Can we wait until 's actual fix for this issue or do we need workaround to reduce this flake?
Let's go with 's fix. I don't think it should take too long


Question: }, }, } waitForQuota(t, quota, clientset) <del> tests := []testCase{ { description: "node port service should be created successfully", svc:         newService("np-svc", v1.ServiceTypeNodePort, true), success:     true, }, { description: "first LB type service that allocates node port should be created successfully", svc:         newService("lb-svc-withnp1", v1.ServiceTypeLoadBalancer, true), success:     true, }, { description: "second LB type service that allocates node port creation should fail as node port quota is exceeded", svc:         newService("lb-svc-withnp2", v1.ServiceTypeLoadBalancer, true), success:     false, }, { description: "first LB type service that doesn't allocates node port should be created successfully", svc:         newService("lb-svc-wonp1", v1.ServiceTypeLoadBalancer, false), success:     true, }, { description: "second LB type service that doesn't allocates node port creation should fail as loadbalancer quota is exceeded", svc:         newService("lb-svc-wonp2", v1.ServiceTypeLoadBalancer, false), success:     false, }, { description: "forth service creation should be successful", svc:         newService("clusterip-svc1", v1.ServiceTypeClusterIP, false), success:     true, }, { description: "fifth service creation should fail as service quota is exceeded", svc:         newService("clusterip-svc2", v1.ServiceTypeClusterIP, false), success:     false, }, </del> <ins> // Creating the first node port service should succeed nodePortService := newService("np-svc", v1.ServiceTypeNodePort, true) _, err = clientset.CoreV1().Services(ns.Name).Create(context.TODO(), nodePortService, metav1.CreateOptions{}) if err != nil { t.Errorf("creating first node port Service should not have returned error: %v", err) </ins> } <del> for _, test := range tests { t.Log(test.description) _, err := clientset.CoreV1().Services(ns.Name).Create(context.TODO(), test.svc, metav1.CreateOptions{}) if (err == nil) != test.success { if err != nil { t.Fatalf("Error creating test service: %v, svc: %+v", err, test.svc) } else { t.Fatalf("Expect service creation to fail, but service %s is created", test.svc.Name) } </del> <ins> // Creating the first loadbalancer service should succeed lbServiceWithNodePort1 := newService("lb-svc-withnp1", v1.ServiceTypeLoadBalancer, true) _, err = clientset.CoreV1().Services(ns.Name).Create(context.TODO(), lbServiceWithNodePort1, metav1.CreateOptions{}) if err != nil { t.Errorf("creating first loadbalancer Service should not have returned error: %v", err) } // wait for ResourceQuota status to be updated before proceeding, otherwise the test will race with resource quota controller expectedQuotaUsed := v1.ResourceList{ v1.ResourceServices:              resource.MustParse("2"), v1.ResourceServicesNodePorts:     resource.MustParse("2"), v1.ResourceServicesLoadBalancers: resource.MustParse("1"), } waitForUsedResourceQuota(t, clientset, quota.Namespace, quota.Name, expectedQuotaUsed) // Creating another loadbalancer Service using node ports should fail because node prot quota is exceeded lbServiceWithNodePort2 := newService("lb-svc-withnp2", v1.ServiceTypeLoadBalancer, true) testServiceForbidden(clientset, ns.Name, lbServiceWithNodePort2, t) // Creating a loadbalancer Service without node ports should succeed lbServiceWithoutNodePort1 := newService("lb-svc-wonp1", v1.ServiceTypeLoadBalancer, false) _, err = clientset.CoreV1().Services(ns.Name).Create(context.TODO(), lbServiceWithoutNodePort1, metav1.CreateOptions{}) if err != nil { t.Errorf("creating another loadbalancer Service without node ports should not have returned error: %v", err) } // wait for ResourceQuota status to be updated before proceeding, otherwise the test will race with resource quota controller expectedQuotaUsed = v1.ResourceList{ v1.ResourceServices:              resource.MustParse("3"), v1.ResourceServicesNodePorts:     resource.MustParse("2"), v1.ResourceServicesLoadBalancers: resource.MustParse("2"), } waitForUsedResourceQuota(t, clientset, quota.Namespace, quota.Name, expectedQuotaUsed) // Creating another loadbalancer Service without node ports should fail because loadbalancer quota is exceeded lbServiceWithoutNodePort2 := newService("lb-svc-wonp2", v1.ServiceTypeLoadBalancer, false) testServiceForbidden(clientset, ns.Name, lbServiceWithoutNodePort2, t) // Creating a ClusterIP Service should succeed clusterIPService1 := newService("clusterip-svc1", v1.ServiceTypeClusterIP, false) _, err = clientset.CoreV1().Services(ns.Name).Create(context.TODO(), clusterIPService1, metav1.CreateOptions{}) if err != nil { t.Errorf("creating a cluster IP Service should not have returned error: %v", err) } // wait for ResourceQuota status to be updated before proceeding, otherwise the test will race with resource quota controller expectedQuotaUsed = v1.ResourceList{ v1.ResourceServices:              resource.MustParse("4"), v1.ResourceServicesNodePorts:     resource.MustParse("2"), v1.ResourceServicesLoadBalancers: resource.MustParse("2"), } waitForUsedResourceQuota(t, clientset, quota.Namespace, quota.Name, expectedQuotaUsed) // Creating a ClusterIP Service should fail because Service quota has been exceeded. clusterIPService2 := newService("clusterip-svc2", v1.ServiceTypeClusterIP, false) testServiceForbidden(clientset, ns.Name, clusterIPService2, t) } // testServiceForbidden attempts to create a Service expecting 403 Forbidden due to resource quota limits being exceeded. func testServiceForbidden(clientset clientset.Interface, namespace string, service *v1.Service, t *testing.T) { pollErr := wait.PollImmediate(2*time.Second, 30*time.Second, func() (bool, error) { _, err := clientset.CoreV1().Services(namespace).Create(context.TODO(), service, metav1.CreateOptions{}) if apierrors.IsForbidden(err) { return true, nil </ins> } <ins> if err == nil { return false, errors.New("creating Service should have returned error but got nil") } return false, nil }) if pollErr != nil { t.Errorf("creating Service should return Forbidden due to resource quota limits but got: %v", pollErr) </ins> } }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --integration test jobs TestQuotaLimitService which was recently in One example run: I suspect this is because TestQuotaLimitService runs through multiple test cases where each test case relies on state created from the previous case. However, resource quota changes take some time to get propagated on the system and the test is trying to validate acceptance of new Services before the resource quota controller applies the necessary changes. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc
/sig network /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Opened a fix here
/triage accepted


Question: checkAPIGroups(t, apiGroup, parsed) } <ins> func TestServiceGC(t *testing.T) { service := discoveryendpoint.NewResourceManager("apis") aggregatedResourceManager := discoveryendpoint.NewResourceManager("apis") aggregatedManager := newDiscoveryManager(aggregatedResourceManager) testCtx, cancel := context.WithCancel(context.Background()) defer cancel() go aggregatedManager.Run(testCtx.Done(), nil) aggregatedManager.AddAPIService(&apiregistrationv1.APIService{ ObjectMeta: metav1.ObjectMeta{ Name: "v1.stable.example.com", }, Spec: apiregistrationv1.APIServiceSpec{ Group:   "stable.example.com", Version: "v1", Service: &apiregistrationv1.ServiceReference{ Name: "test-service", }, }, }, service) require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) // Lookup size of cache getCacheLen := func() int { aggregatedManager.resultsLock.Lock() defer aggregatedManager.resultsLock.Unlock() return len(aggregatedManager.cachedResults) } require.Equal(t, 1, getCacheLen()) // Change the service of the same APIService a bit to create duplicate entry aggregatedManager.AddAPIService(&apiregistrationv1.APIService{ ObjectMeta: metav1.ObjectMeta{ Name: "v1.stable.example.com", }, Spec: apiregistrationv1.APIServiceSpec{ Group:   "stable.example.com", Version: "v1", Service: &apiregistrationv1.ServiceReference{ Name: "test-service-changed", }, }, }, service) require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) require.Equal(t, 1, getCacheLen()) } </ins> // Test that a handler associated with an APIService gets pinged after the // APIService has been marked as dirty func TestDirty(t *testing.T) {

The in apiserver grows infinitely without ever purging serviceKeys which are not referenced by any APIServices. /sig api-machinery /cc /assign a limited size cache APIService your services Spec.Service field to a new address step 2 and watch the list grow No response_ <details26-1.27 </details<detailsN/A </details<details</details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/triage accepted
/assign
what do we want to the limit the cache size by?
the cache currently doesn't purge s which are not being used by an
is this still an issue? should it be resolved before enabling by default?
Can you elaborate? I am not sure what you mean by enabling by default?
Yes That's the cache for the downloaded discovery for each apiservice. The list probably doesn't grow very large. Fix is in . Considering the pain of non-aggregated discovery on some users, and the time they have had to wait to get the feature, I'd say the trade-off is that this isn't nearly as bad.
It's also worth noting that the cache should only grow for Aggregated APIServices that themselves expose aggregated discovery endpoint. We don't store a cache entry here otherwise. The list doesn't grow unless the aggregated discovery document was successfully fetched. This is a very rare case and especially at release a minuscule number of people on the bleeding edge might run into it. And even then it is not likely to be changing the so often once it is working enough to store a cache entry.


Question: <ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. cd "${KUBE_ROOT}" # generate json spec -> yaml test/conformance/gen-conformance-yaml.sh # replace checked-in yaml cp _output/conformance.yaml test/conformance/testdata/conformance.yaml </ins>

The currently provided command for generating requires bazel, the command should be updated to one which does not require bazel Currently: Suggested: /sig testing /sig architecture /area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
We're going to need to update this to verify under as well instead of under the bazel "unit tests", ref: I'll work on this.
done in
This is wonderful to see. Thank you


Question: hostsFileContent = managedHostsFileContent(hostIPs, hostName, hostDomainName, hostAliases) } <del> return os.WriteFile(fileName, hostsFileContent, 0644) </del> <ins> hostsFilePerm := os.FileMode(0644) if err := os.WriteFile(fileName, hostsFileContent, hostsFilePerm); err != nil { return err } return os.Chmod(fileName, hostsFilePerm) </ins> } // nodeHostsFileContent reads the content of node's hosts file.

What happened: Process running as non-root will fail to work as it cannot access /etc/hosts inside the container when umask of kubelet is configured as 027. It is because the ioutil.WriteFile() used to create /etc/hosts is impacted by umask. Even though kubelet specify the permission of 0644 explicitly, with umask of 0027, it will be configured as 0640. As a result, application running inside the pods with non-root user, failed to access it. What you expected to happen: kubelet should pre-configure required umask by itself, instead of depending on the environment. At least we should document the required umask. How to reproduce it (as minimally and precisely as possible): Lauch kubelet with umask of 027 Anything else we need to know?: It's the same problem figured out. Environment: Kubernetes version (use ): 1.19 Cloud provider or hardware configuration: N/A OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/area kubelet
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with .

Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Expect(c.Deployments(ns).Delete(deploymentName, nil)).NotTo(HaveOccurred()) }() <del> waitForDeploymentStatus(c, ns, deploymentName, 3, 2, 4) </del> <ins> err = waitForDeploymentStatus(c, ns, deploymentName, 3, 2, 4) Expect(err).NotTo(HaveOccurred()) </ins> } func testRollingUpdateDeploymentEvents(f *Framework) { ns := f.Namespace.Name c := f.Client // Create nginx pods. <del> podLabels := map[string]string{"name": "sample-pod"} </del> <ins> deploymentPodLabels := map[string]string{"name": "sample-pod"} rcPodLabels := map[string]string{ "name": "sample-pod", "pod":  "nginx", } </ins> rcName := "nginx-controller" _, err := c.ReplicationControllers(ns).Create(&api.ReplicationController{ ObjectMeta: api.ObjectMeta{

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: // Total number of available pods (ready for at least minReadySeconds) targeted by this StatefulSet. // This is a beta field and enabled/disabled by StatefulSetMinReadySeconds feature gate. <ins> // +optional </ins> AvailableReplicas int32 `json:"availableReplicas" protobuf:"varint,11,opt,name=availableReplicas"` }

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <del> apitypes "k8s.io/apimachinery/pkg/types" </del> utilerrors "k8s.io/apimachinery/pkg/util/errors" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/sets"

Integration test timed out due to the following error The error appeared 748 times in the log. Looks like lots of churning happens during the apf configuration bootstrapping phase. Is there a race condition between when the type is being registered and the API calls? Stack trace: Integration job link: PR: I searched for this error in CI and this PR, looks like we only see this error in this PR so far. So I am hoping it has not introduced any flake. (maybe we need to let more time to pass to start seeing flakes) We should not see this error I saw it only in the PR mentioned.
/sig api-machinery cc
I see
The fact that it happens to a non-mandatory object implies that the object was received from the server, so the type must be in the server. The APF config-consuming controller does this:
Look for the output from this line:
Another PR with this failure:
the panic means the request scope FieldManager is nil that is initialized here: OpenAPIModels is set here: called from here: and openAPIModels is constructed here: it looks like it is possible for openapiconfig to be unset, which causes the openapi models to be nil, which causes apply to panic. I don't know if it is only the integration tests that are subject to that configuration. cc /priority important-soon (since this is now being exercised in bring-up of integration test servers)
if APF is creating/bootstrapping the objects without using apply, and then updating status using apply, that would explain how the object exists but apply fails
I suspect that APF is not successfully updating status at all in the integration tests with this server configuration
do you want me to revert , while we work on a solution?
I don't know how bad the test impact is. Work with / to determine if we should revert or fix forward
: I do not follow why you talked about server-side stuff then suggested the problem is in the client (a controller in the kube-apiserver) not supplying a fieldManager. The client does supply the fieldManager, for both creates and updates. See the following lines:
Different field manager. is set server-side in the endpoint handler based on the server configuration.
So if the problem is server-side, why did you suggest the problem is in the client?

Regarding the client, I looked in and found this line: actually several of the same form, all with the same fieldManager.
I don't think I did suggest that :) all of the setup and calls in are server-side
: please help me understand . That is about the client side. Are you saying that a client that creates an object using and later updates the object using --- with a non-empty fieldManager in both --- will run into this server-side scenario? If so, is that because this client behavior is wrong or because there is a bug server side?
That was in response to , trying to figure out how the object was created but could not be updated. Yes, when the server is not configured to enable openapi and therefore does not have apply enabled. Because the server is misconfigured (which, from what I can tell, is limited to some of the server instances started for integration tests)
Looking at and it is not obvious to me whether or how openapi could be disabled. Would that be by turning off the feature flag? FYI, the inclusion of "V3" in the flag makes me unsure what the server does about other versions. : I think you are saying that it is possible to disable openapi and that this in turn disables server-side apply. That seems kind of surprising to me. Does this mean that it is simply incorrect to run a client that does server-side apply with a server that has openapi disabled? Is this intended? Is it documented anywhere?
It's not intended as an externalized configuration option. integration tests have the ability to fiddle with many of the config struct options in the server that a kube-apiserver binary invocation could not. They intentionally use this to do things like inject specialized authorizers / authenticators for testing purposes, but also can unintentionally result in a test server that doesn't match a real API server invocation. I think this is an integration test server configuration problem, which should be fixed.
is correct that we have a misconfiguration in the integration tests and SSA is only enabled on a subset of integration tests. I'd like to understand the test failure conditions a bit better though. The can be consistently reproduced on a non SSA targeted test. (Eg: selected from one of the test failures). However, I'm seeing the panic message while the test is still green and passes locally.

it's a bit bizarre what contributes to the flakiness of the test because the FieldManager panic can be reproduced 100% of the time.
The panic occurs when APF is updating status (which it just switched to do using SSA). The APF status update failure does not block the API requests the test cares about, so the test passes. I could see extra CPU usage contributing to flakiness.
Thanks for the explanation. Created to fix the integration test configurations.
/triage accepted
The integration test job started timing out dramatically last night (50% of runs) once merged. Since that was the only change made to integration tests recently, it seems related. I think we should revert (and therefore ) until the test impact is understood
revert of -
did this get fully reverted? we're still seeing panics in integration tests over this:
bumping priority and milestoning until we know we're not currently triggering panics
looks like it got reintroduced in , revert open in
For integration tests, maybe we can cheat and cache the OpenAPI data, so that it's not recomputed for each test?
:-/ maybe... but I really don't want to have non-real init paths in integration if at all possible
I wish we could go a little further in lazily initializing openapi (especially v2), so let's see what we can do here (after code-freeze probably)
if SSA is actively using the openapi, it being lazy won't help, right?
does the revert mean that we can remove milestone v1.24 from this issue?
Yes, cleared
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Taking a fresh look at this PR and wondering if it has been mostly fixed already? From comments my understanding is that this panic occurred due to two conditions: running as part of the integration test suite is asked to service an SSA request while are nil. The apiservice has logic to panic in this case. From the code path commented is nil when the server is created with is nil. APF worker (running alongside an integration test) issues an SSA request to the test apiserver. Since the apiserver is misconfigured it panics. This panic could happen sporadically because sometimes the tests finish before the APF worker is able to issue the request. As a brief check to see if this might be fixed, I ran the test mentioned was failing from SSA. I logged out the 's created and found that they all had managed fields. I saw this as an indication SSA was now functioning properly. Indeed, to start the test server the function calls . This function has a code path which unconditionally leads to the apiserver's being set. calls before running the returned apiserver: calls before using the returned config to create the returned apiserver: uses which sets : Thus, any test using should no longer be affected by this panic. The question then becomes: do all integration tests start their servers in this way? It's hard to say. I found a PR (, ) which refactors many occurrences of the old stanza: Into something using the new method: ` But it appears is still being used in a few integration tests: Should these occurrences be refactored to use too?
There was a ton of changes I did around that. But the last PR (already removing the old is already out for review): - FYI Thinking about it now, I guess I agree that the problem might have been 10 different way kube-apiserver was set up through different tests.

I just don't know why it was causing flakes, not failing consistently...
There were definitely test setups that were not enabling openapi in the recent past. Two possibilities for flakes: Enabling openapi in all integration test setups was expensive and caused spurious timeouts The excessive numbers of background panics handled and logged in integration tests when the API server tried to handle SSA requests bogged down the CI machine (I think something like 40,000 panics with full stack traces each were getting logged)
i don't think that's the case. With the recent changes that I was doing as part of I think OpenAPI is already enabled for all tests having kube-apiserver (almost all) and that seems to work. But you remainded me that back then we also had an issue that we were running many more tests simulanuously, which could have been contributing to this problem significantly. The second reason is definitely possible contributor too. I guess my concern was that if some requests were panicing due to lack of OpenAPI, this means that these aren't really needed. But that actually might make sense - the fact that we weren't able to upgrade status of PriorityLevels/FlowSchemas shouldn't prevent non P&F-tests from succeeding, and that worked in P&F tests...
I think the panics were happening in async controllers, and no functionality blocked by the controllers intersected with the surface being tested, it just (maybe) caused log/perf issues
I believe has resolved the panic issue, I was able to verify with which changes the priority & fairness controller to use server side apply. Before merged, I analyzed the integration log: Integration test log file from an unrelated pr does not contain any trace of the panic as expected: On the other hand, the log file from integration job for (before merged) had plenty of these panics even though the integration job turned green: I wanted to check which tests produced these panics, so I did a breakdown by package Then I narrowed it down to the individual test(s): Before merged, the above tests used , and either is or is which results in to be nil. But a quick code search revealed that many more integration tests used , so why were the other tests not affected by the panic?

I inspected the log of a test that was not affected by the panic, and grepped for , and found the following: In the above case, the apf controller had not advanced far enough, the controller was active for about , may be these tests finish quickly and signals the server to shutdown before the apf controller can send an APPLY command. On the other hand, log from a test that was affected by the panic shows that the apf controller was active for more than , and has lot more activities: I am assuming that integration tests have at least log level 4. Also, the size of the log file differed significantly, without panics around , and with panics over and changed all these tests to use and now integration log for shows no panic.
Great! (This also proves why eliminating gazilion different things of initializing stuff in favor of one makes sense). - FYI ^^ - let's merge the PR then - I just one question there, so let's wait for Jordan to respond it.
Can we just force the openapi/fieldmanager to be set at start-time now that SSA is GA, it should basically always be present, rather than fail when SSA is unavailable (that's probably wrong by the way). If that triggers any test flakiness, then we would solve these. We can wait for to be solved since it will almost certainly have a huge impact on this.
Has this been done Jeff?


Question: concurrent-serviceaccount-token-syncs concurrent-service-syncs config-map <ins> config-map-data </ins> config-map-namespace config-sync-period configure-cloud-routes

Kubernetes version : What happened: When I try to display configmap content with yaml output (kubectl get configmap webapp-inifile -o yaml), the output is sometimes pretty, and sometimes not -depending newlines are effectively displayed or not - just n char- I can't find what's the difference between some file and some others. (limit on numer of lines ?) Configmap is created with --from-file option Ex : What you expected to happen: More human readable display, in order to be able to easily change parameter when kubectl edit congifmap How to reproduce it :
Hrm, this seems to me like the right answer is to special case 'kubectl edit' for configmaps, rather than try to hack around with the YAML decoder. I suppose we could use block quotes (e.g. ) if we see newlines but that seems error prone...
Closing this, pretty print is there, kubectl configMap edit support is there.
This still happens on kube 1.7 :-( I'm not 100% sure how to reproduce, but what I did was: have an existing configmap and edit a few lines exit the editor ... now I see the characters instead of actual newlines. even shows the characters
I'm also seeing this issue in kube 1.7. During the past few days I have edited a configmap ~110 times, and seen the formatting get destroyed (spaces messed up, characters visible) twice. I can't determine anything specific that I did in those two edits that would have triggered the formatting messup.
I'm still seeing this issue too. Pushed a lot of identical configmaps, some ones are displaying good, and some other bad. For example, I have a configmap with 3 files in ; they all are identical except one variable ; 2 of them display normally, and the last one displays with n ...
also seeing this issue on 1.7
can we reopen this issue? or open a new one?
I created a brand new configmap using and when I viewed the yaml with it's full of n characters and escape characters. This makes it very difficult to edit.
I am experiencing the same. n character is messing up a configmap I'm creating with --from-file option. Anyone found a reason why this is happening ?

Also failing in 1.8.4
I am also facing the same issue of formatting .. I have a application configuration of 400-500 lines. Tryied many times. For fist 40-50 lines the format is good ans easily readable.. However, when i try to create config for entire configuration file its very difficult to edit the config map..
Same issue here. No idea what makes one configmap show better than another.
This issue corrupted my aws-auth configMap that authenticates my user to access the EKS cluster and while trying to add other users to the configMap the formatting corrupted the configMap file and now I am locked out of my EKS cluster. Version: 1.10.3
I don't know if this will help somebody else but I was facing the same issue with escaped data in a config map edit and get (the config map had a single .conf file), and it seems that I had a unneeded extra space at the end of a single line in my data. When I removed that extra space the pretty printing was maintained in edit and get.
11 Still same
still same ( there is an example if install helm install stable/prometheus , you will find cm/prometheus-server )
I agree with - I can recreate it every time If you were to execute the following (ensuring no trailing spaces), the configmap remains "pretty" However, repeat the above (or edit the existing one with ), and put a trailing space after one of the key/value pairs, and that then messes up the configmap contents
Can we reopen this issue? I need this feature to change the yaml in cm, and I have been struggling with the intent and n for several days ...
Please reopen this ticket, it's incredibly frustrating to have to deal with this even on small configmaps
Got this also. I had to remove tabs from my config file to get the yaml multiline display.
Just created a messed up configmap on kuberntes 1.12.2.
Likewise.
OMG thank you!
Just ran into this in EKS (1.10.3) -- highly annoying. I have one config map that will come out OK if I use create directly, but if I pipe the output of to apply then it gets munged. Does seem like this is something that needs addressing.

Have the same issue and fixed by removing tabs
Have the same issue, don't have any tabs in file. This issue totally should be reopened.
This issue is valid and I am not sure why it was closed
Not only spaces. Trail symbols too. I uses vim for it
"n n" (spaces between end of lines) in one of my files also messed up the pretty display.
Created a new issue when I hit this again and was able to repro using the trailing whitespace
I've removed all the trailing whitespaces and tabs. The problem is gone.
Check any possible special chars. It solve my problem some time ago
I have seen this issue in K8 1.7 -1.13 kubectl create configmap NAMESPACE --from-file=$PATCHFILE --dry-run=true --output=yaml where PATCHFILE is attached file below. I renamed it from .json to .log so this webpage would let me include it.
Solve the issue by delete whitespace at each end of line.
The output of is ugly. Wether using json or yaml, k8s injects a whole bunch of escape characters and messes up the formatting of the original config file. It would be very nice to have a switch that tells it to dump the config out "normally"!
I a new line in a configMap using Tab for identation. After changing to Spaces instead of Tab, I was able to see the configmap as expected...
Same here, white space/s at the end of line ruin the output, everything back to pretty when white space/s removed.
This worked for me.
Since I've still met this on 1.16, I'll point a bullet list of the regular culprits for the configmap messing: spaces before newline:Tabs:Special characters like ãẽõ, áéó, öß will also mess your files. Look up for them in your comments, they might pass through!
An easy one liner to remove trailing whitespaces in a ConfigMap:
Please reopen. This issue still remains. This format makes eks aws-auth configmap very hard to maintain. Once the file is applied with trailing whitespaces the whole thing gets lots of n and . As a temporary measurement is there a way to reformat the configmap in the "readable" way? after the n are created?

if you use way didn't reformat the file for me.
Nurza's method works. But you may have a space before EOF as well: Or you may have CR's in there , in which case you should remove the CRs. Or you may have a TAB, which you should replace with one or more spaces.
No CR/TAB/new lines. I have a "clean" file that after I apply it all works great. (display as expected) After I add one trailing white space to it, it gets those n. I tried to run Nurza's method but it didn't work. Any other Ideas why Nurza's method is not working for me ? This a part of the output I'm getting with
has a space after it, although you don't see it. Because the LF is on the next line, there must be a space before it.
Correct but if I run Nurza's method the n are still there. Am I still missing anything? This is what I run
Nobody said that sed expression was perfect. Once you know this is your doing -- and you know how to fix it -- you're not bothered by this behaviour anymore.
This is scary shit :hankey: ... what is the industry practice to edit the file like these: aws-auth...? I managed to type 20+lines by hand :disappointed: and it worked on 3rd attempt :disappointed_relieved:
Cant' say about the 'like this', but aws-auth you can use eksctl to do the hard work for you:
If the formatting of cm goes wierd a simple hack to get it back to normal is : kubectl get cm configmapname -o yaml cm.yamlNow copy the contents of file and past it on is powerful tool to check the linting of yaml files. This will provide you with the configmap as expected with correct formatting. Paste the output in another yaml file (for e.g - )kubectl apply -f
Anyone knows why the issue is closed? It's not fixed as far as I can tell.

If you just want the raw output as it was read in when created --from-file, you can use jq to get the raw string (without escaped newlines etc) If you created a configmap from a file like this: Get the data:
Edited a cm, came back in, and had n all over. Something about white space in my previous edit? Not great overall :/ On kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Awesome, very good website~
Thank You very much
I was able to fix this behavior by: 1) Don't use tabs, convert to spaces 2) To remove spaces before a newline character, use this: It will also convert CRLF to LF only, by removing .
Something small that exacerbates this issue is that in python's standard lib module , up to some version (I think python 3.4, see ), pretty printing adds a trailing space after every comma by default, unless you supply a list of separators yourself that omit the spaces (e.g. like ) So if you have old-ish glue (e.g. python 2.7) that parses a k8s configmap into a file for archival purposes, and you made that file readable by pretty-printing it, using that file to regenerate a new configmap results in this gross escape behaviour.
Hello There, One of our colleague has edited configmap with wrong data, now whole cluster inaccessible with any of role, need help on immediate basis and much appreciated. Error: through eksctl - Error: getting auth ConfigMap: Unauthorized through kubectl error: You must be logged in to the server (Unauthorized)
yaml marshallers will write a string to a multiline string e.g.

UNLESS string contains a character string terminates in a space character string contains a substring which matches the regex
I am in the same situation. Were you able to solve this?
Also reproduce in v1.22.3 :(
I occasionally encountered this problem in v1.15.6 or v1.21.2, but eventually I was able to deal with it by using edit and then using vim's own set list to back-propagate the original static configmap code configuration under gitlab to see if there were extra spaces, and then modifying the local configmap file based on the online configmap's set list display under edit. The state of the online configmap under edit, to modify the local configmap file, is ultimately able to deal with this problem, but if there is a better official way to do it would be good.
yes if you are working on linux
i would like to share an observation that helped me sort out the line merging and n. it appears to be related to the last line of the file you are using (--from-) if the last line ends with empty space or tab. the config map is merged up in one line with n as line feed if you remove the empty space from the last line, the ConfigMap will import file as is with +
I got the same error after editing the ConfigMap file. I used an online tool to remove the trailing white space that we cannot see in the text. After copying the content which removed trailing whitespaces I was able to resolve this issue FYI - (online tool which I used)
in v1.24 also i can see same issue. how we can edit cm with many lines.
Finally, I found that the absence of a value defined or the value is empty string in will cause the issue. data: {{ }} is not defined in my yaml just use quote to the value resolved
delete space before n or after n it works for me
did anyone find solution when we edit cm how we can see in proper format as for me really difficult to edit cm as we have many n char and I can still see this issue in k8s version 1.28


Question: <ins> // +build freebsd linux /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "testing" "time" ) func TestGetBootTime(t *testing.T) { boottime, err := GetBootTime() if err != nil { t.Errorf("Unable to get system uptime") } if !boottime.After(time.Time{}) { t.Errorf("Invalid system uptime") } } </ins>

What would you like to be : Currently, does not have unit test coverage. Specifically, there's no unit test coverage for . I'd like to see unit test coverage for . See for a sample of how to write platform specific unit tests. Why is this needed: There's a fairly small chance of being modified, and thus breaking, in the future without this test coverage. However, it should be relatively straightforward to add the test coverage, and could make a good first issue for a new contributor. There's also the benefit that it helps keep the test coverage in high :) And we never know, it might help catch a bug some day :) Extra note I think this ticket could be a great candidate for someone who is new to contributing to sig-node/k8s. I'd prefer to please give first dibs for picking this up to any folks who are currently underrepresented in the k8s community. Please reach out to on k8s Slack (or post directly in #sig-node and tag me) if you're interested, and I'll assign you the issue. I'm happy to mentor and be helpful in whatever way I can :)
Assigning this to Ali Farah (" (Ali, for some reason it won't let me assign you - can you assign yourself?) /sig node
Thanks ! /assign


Question: if [[ "${ENABLE_DEFAULT_STORAGE_CLASS:-}" == "true" ]]; then setup-addon-manifests "addons" "storage-class/gce" fi <del> if [[ "${FEATURE_GATES:-}" =~ "CSIDriverRegistry=true" || "${FEATURE_GATES:-}" =~ "CSINodeInfo=true" ]]; then </del> <ins> if [[ "${FEATURE_GATES:-}" =~ "AllAlpha=true"  || "${FEATURE_GATES:-}" =~ "CSIDriverRegistry=true" || "${FEATURE_GATES:-}" =~ "CSINodeInfo=true" ]]; then </ins> setup-addon-manifests "addons" "storage-crds" fi if [[ "${ENABLE_IP_MASQ_AGENT:-}" == "true" ]]; then

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: Now, how to run the entire application in Kubernetes? <del> To simplify running this application, we have a single file, k8petstore.sh, which writes out json files on to disk.  This allows us to have dynamic parameters, without needing to worry about managing multiple json files. </del> <ins> To simplify running this application, we have a single file, [k8petstore.sh](k8petstore.sh), which writes out json files on to disk.  This allows us to have dynamic parameters, e.g. the namespace is configured by `NS` whose default value is `k8petstore`, without needing to worry about managing multiple json files. </ins> You might want to change it to point to your customized Go image, if you chose to modify things, like the number of data generators (more generators will create more load on the redis master).

During the review of , it's better to also update and to use non-default namespace. cc .
cc
/ , I can continue to work on this one :).
/ , can you help to assign this issue to me? So I can trace the tasks in my hand :).
I'd be happy to assign you once you accept the invitation to the org. Do you plan to work on this now? If you're not actively working on it, I'd prefer to leave it unassigned.
, would you help to send the invitation ? Yes, I'd like to work on this now, that's a good start for me to learn example & e2e test suite :).
Already sent. Visit to accept. See
, I'm kuber now , thanks. I'm working on this now :). Checked and , openshift-origin is not using default namespace now; and we can update 's script to update namespace. But it seems and other example are still using default namespace, should we go through all example and update them accordingly?
, should I update all ns in one commit or one commit per example?
In which examples do you plan to make this change? cc
Making this change in some examples would be useful, but only if we explain why we're doing it. Additionally, linking to existing docs and blogs about namespaces would be helpful: I don't want to do this in all of the examples, because it does add an extra step, and labels also provide a means of managing a group of related resources within a namespace. Perhaps could recommend examples where this would make sense.


Question: if err := runtime.DecodeInto(codec, buf.Bytes(), &cmList); err != nil { t.Fatal(err) } <ins> if len(cmList.Items) != 2 { t.Fatalf("Expected 2 items in the result; got %d", len(cmList.Items)) } if !strings.Contains(string(cmList.Items[0].Raw), "key1") { t.Fatalf("Did not get first ConfigMap at the first position") } if !strings.Contains(string(cmList.Items[1].Raw), "key2") { t.Fatalf("Did not get second ConfigMap at the second position") } </ins> } func TestRunApplyViewLastApplied(t *testing.T) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: if securityContext != nil { allErrs = append(allErrs, validateHostNetwork(securityContext.HostNetwork, spec.Containers, specPath.Child("containers"))...) <ins> if securityContext.FSGroup != nil && !validation.IsValidGroupId(*securityContext.FSGroup) { allErrs = append(allErrs, field.Invalid(fldPath.Child("fsGroup"), *(securityContext.FSGroup), IdRangeErrorMsg)) } if securityContext.RunAsUser != nil && !validation.IsValidUserId(*securityContext.RunAsUser) { allErrs = append(allErrs, field.Invalid(fldPath.Child("runAsUser"), *(securityContext.RunAsUser), IdRangeErrorMsg)) } for i, gid := range securityContext.SupplementalGroups { if !validation.IsValidGroupId(gid) { supplementalGroup := fmt.Sprintf(`supplementalGroups[%d]`, i) allErrs = append(allErrs, field.Invalid(fldPath.Child(supplementalGroup), gid, IdRangeErrorMsg)) } } </ins> } return allErrs

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: klog.V(4).InfoS("Validating the CRI v1 API image version") r.imageClient = runtimeapi.NewImageServiceClient(conn) <del> if _, err := r.imageClient.ImageFsInfo(ctx, &runtimeapi.ImageFsInfoRequest{}); err == nil { klog.V(2).InfoS("Validated CRI v1 image API") } else if status.Code(err) == codes.Unimplemented { return fmt.Errorf("CRI v1 image API is not implemented for endpoint %q: %w", endpoint, err) </del> <ins> if _, err := r.imageClient.ImageFsInfo(ctx, &runtimeapi.ImageFsInfoRequest{}); err != nil { return fmt.Errorf("validate CRI v1 image API for endpoint %q: %w", endpoint, err) </ins> } <ins> klog.V(2).InfoS("Validated CRI v1 image API") </ins> return nil }

It "connects" to dockershim There is no Run crictl v1.26.0, without setting up (or using -r) Worked in crictl v1.25.0, that still had CRI v1alpha2 This patch fixed the issue, to make the fallbacks work: <detailsv1.26.0 </details<detailsLima </details<detailsPRETTY_NAME="Ubuntu 22.04.1 LTS" </details<details</details<detailsv1.6.12 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I agree that we can check for unavailable connection because it will only be done on kubelet startup.